Subject: Digest for the period 1/20/2004 - 1/21/2004 Date: Wed, 21 Jan 2004 01:00:27 -0500 Table of contents ---------------------------------------------------------------------- 1. Exam scoring (Jeremy Bergsman) 2. Exam guide (Gordon Strong) 3. Re: Exam heresy (BillPierce`at`aol.com) 4. Re:More Heresy (Pete) (Steve Casselman) ---------------------------------------------------------------------- From: Jeremy Bergsman Date: Tue, 20 Jan 2004 02:07:54 -0500 Subject: Exam scoring I thought I would throw in a few comments on exam scoring, since these things always seem like a big secret until you've done it. It's a lot like beer scoring. How many points off do you take for an IPA that isn't quite as bitter as you'd like? How do you reward that plambic that just nailed the whole barnyard/estery thing? These are essay questions and scoresheets--there's no easy way to decide in advance how to mark an answer. When you get your exam score you get several pages back, the first page has the score (broken down into several categories), some suggestions for further study, and a section on "interpreting your score." The main part reads like this: =============================================================================== <60: Poor knowledge of brewing and/or styles with insufficient communication skills to be a judge. Generally has weak tasting skills that need development. 60s: A basic grasp of fundamentals. May have some big knowledge gaps, but has the minimum acceptable communication and judging skills. 70s: Knows basis well enough to reach the certified level, but there are some errors and gaps in the answers. Depth in answers is not necessary. 80s: Good knowledge of all subjects, with minor errors but no significant gaps. Some depth, and the essay and tasting portion should show similar ability. 90s: Excellent knowledge level with no significant errors or gaps. Good depth to answers with evidence of independent thought. Excellent tasting and communication skills. =============================================================================== Just like a new judge has to learn what makes for a "very good" beer, a new exam grader has to learn what makes a national judge. Of course we should already have a pretty good idea about this. BTW, as was said previously, on my exams I am writing about a full page of 10 point single spaced replies in addition to the first page described above. I would like to relate a personal anecdote about exam grading. It's very late and I will probably regret writing this, but.... In large part I agreed to become an exam grader because I had the perception that grading was too hard and good judges were getting low scores onthe exam. Now that I have done it for a few years I'm not so sure. I'm sure that test anxiety prevents some people from realizing their potential, but I am amazed at many of the exams I grade. They just aren't that good. I generally feel like I'm bending over backwards, giving the benefit of the doubt wherever I can. As I'm sure most of you know, on most of the questions you have to describe one or more beer styles. Believe me, I'm not marking people down for getting the OG off by 3 points. MOST people fail to describe one of the major aspects of a beer (like hop character) for MOST of the beers. A good fraction of the examinees is unable to describe at least one of the beer styles AT ALL. You wouldn't believe how few of your fellow judges know the difference between Berliner Weiss and Bavarian Weizen. We're not asking that you memorize all the numbers, but you should be familiar with all the styles! Whole parts of the answers are often left out, like the required commercial examples, or when it asks to compare the beers--they are not compared--just described. This is all probably going to make people who feel they should have gotten a 90 even more upset, but all I can say (since I've never seen an exam that could be construed by anyone to be a Master level exam) is if you know the styles and you can sit down and practice writing answers that just methodically cover all the aspects of a beer (like you are judging it: aroma, color...) and actually answer the questions, you are basically in National territory. The only thing you should have to memorize IMO is the commercial examples. If you read the examples of good answers I think you can see they are achievable. I was unhappy with my exam score. Looking back at it I have a feeling my exam wasn't as good as I thought when I turned it in. Nobody likes working that hard and not doing well. If I didn't know it would be a logistical nightmare to return the exams I would say it would be good, because I think some of the complaints would go away when people could review their exams in the cold light of day. None of this is directed at anyone who posted here, since I of course have no idea whether I've ever seen your exam, but this is my experience in the program. -- Jeremy Bergsman jeremy`at`bergsman.org http://www.bergsman.org/jeremy ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: Gordon Strong Date: Tue, 20 Jan 2004 08:39:14 -0500 Subject: Exam guide I'm also reading this thread with interest. I'd like to add a few points and provide some of the requested information. First, I do agree with some of the sentiments that it would be helpful if there were standardized deductions for common errors or at least more common examples of the level of detail required to obtain full credit (or varying levels of partial credit). Perhaps this is something the BJCP will take up in the coming year. As it is, the burden falls on the graders (and the reviewers) to assess the question in its entirety and make a holistic estimate. While this might give some variability on individual answers, when looking at the entire test the final level does tend to be more closely evaluated as to the ranking it grants. When a final score is borderline between levels, the reviewers pay careful attention to the full test to properly place it in the most appropriate rank. This point may be obvious but grading essay exams is inherently subjective, and the process of testing is inherently critical. Tests are meant as evaluation and diagnostic tools. The process of taking a test might be humbling, but I contend that it's a reasonable way of determining knowledge. There might be more accurate ways of determining knowledge, but it's hard to do better given the limitations of finding suitable exam administrators, proctors and graders. I agree with Pete Garofalo that it becomes apparent when grading exams when a person really 'gets it'. And I also agree that the quality of score sheets I get back from competitions does tend to reflect the level of the judge. For those asking about more detail on how the exam is scored and how to do well on the exam, I suggest you take a look at the presentation I gave at last year's AHA Conference in Chicago. Check out http://207.89.201.107/Presentations/Mastering%20the%20BJCP%20Exam.pdf. It does include a real-world example of the difference between an answer given a 7/10 vs. a 10/10. It also contains my lessons learned for doing well on the exam, including study, preparation, and test-taking skills. From the feedback I got, it was well-received although others who used the material might want to offer their own opinions. I've graded over 250 exams since 1997, administered or proctored 7 exams, taught 5 judge prep classes, and been an associate exam director for the last 2 years. I've judged in about 100 competitions, and entered at least that many. I have observed judges throughout the entire process and have had many discussions with judges on the pros and cons of our system. I don't think it's perfect, but it's also not as bad as some are making it seem. As you can see from the replies, the people involved with exams and exam policy at the BJCP do follow this list and do pay attention to the discussions. The process of highlighting deficiencies and offering suggestions for improvement is welcome (although the tone of the critique also does matter). Regards, Gordon Strong ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: BillPierce`at`aol.com Date: Tue, 20 Jan 2004 10:32:31 EST Subject: Re: Exam heresy I have been noting with interest the recent debate about the exam structure and grading. I believe it's healthy to discuss these issues because it promotes a clearer understanding of the expectations of the BJCP and encourages objectivity and fairness as well as accurately reflects the skills necessary to evaluate and compare beers and provide useful information to brewers. I agree that the rote memorization of the specific numbers for each style and substyle is of somewhat limited benefit. Isn't it sufficient to be able to state that a Special or Best Bitter, for example, is a low to moderate gravity English ale of medium gold to copper color, with bittering ranging from moderate to somewhat pronounced, and balanced by malt flavor, often with a little caramel sweetness in the finish? Does being able to recite the numbers (O.G. 1.039-1.045, F.G. 1.009-1.014, IBUs 20-45, SRM 6-14, ABV 3.7-4.8%) indicate that the exam taker really knows any more about the style or merely that he/she is good at parroting statistics? At every good competition (the great majority I have judged) either the organizers provide a copy of the style guidelines at each table, or at least one of the judges at each table brings a copy with him/her. Prior to the evaluation of the calibration beer, the judges read aloud the and briefly discuss the guidelines for the style, often mentioning related styles and those with which it may be compared and overlap. At that time the specific numbers and ranges are stated. Several months ago I retook the exam (I'm currently Certified) in hopes of improving my score. My knowledge of brewing and styles has certainly improved since my first exam in 1999. I have since attended the Siebel Institute, I spent some time brewing professionally and I moderate an online discussion forum (the HBD's Brews & Views) that answers a lot of homebrewing questions. I'm not encouraged when I hear that Mike Dixon, for example, did not achieve Master level on his retaking of the exam. I don't mean to single Mike out, but I know of his numerous posts to the rec.crafts.brewing newsgroup for many years; his knowledge of homebrewing is of the highest caliber in my opinion. Now I realize that brewing knowledge is only part of the exam. Highly specific knowledge of beer and the style guidelines is also required. But my point is that I am extremely confident of his ability to evaluate beer and provide detailed information about it. I know that he understands what is necessary to brew high quality beer and how to improve it. I have heard various pieces of information over the years about the range of exam scores. I have no way to verify them, so I am mentioning them here. Among these, for example, are that 40 percent of exam takers fail (that is, score below 60) and that only 4 percent score 90 or above. Another "rumor" is that there are a much greater number of scores at the top of each level (for example, 67-69 and 77-79) than at the bottom (e.g., 60-62 and 70-72). The inference is that the judges like to offer a carrot to encourage retaking of the exam, in order to suggest that with only a small amount of study it would be possible to raise the score to the next level. This is analogous to what teachers call the "plus syndrome," that is, there are often a greater number of B+ grades given than B- grades. This provides a pat on the back to students without any substantive benefit, as the grade is rounded to the letter anyway. I would very much like to see a published distribution of the scores, to let exam takers know how the scores fit over a range. I also agree that it would be very beneficial to publish actual specific examples of answers that have been graded and given a particular score. I would like to know what is necessary to receive a score of 3, 5, 7, 9 and 10, for example. This would be of great use both to prospective exam takers and graders. Additionally I will mention that the information about the published times for exam scoring and return needs to reflect the actual performance of the scorers. It has been four months now (nearly 18 weeks) since I took the exam in September, yet I still have not received my score. Let's continue this discussion. I believe it's very important to maintain the image of the BJCP as a standards organization that reflects what is truly necessary to achieve its stated goals. -- Bill Pierce Cellar Door Homebrewery Now located in Burlington, Ontario ---------------------------------------------------------------------- From: Steve Casselman Date: Tue, 20 Jan 2004 11:37:28 -0800 Subject: Re:More Heresy (Pete) Of course you would have your choice of what test to take. Are you really concerned about snobbiness over which test you took? Steve > ---------------------------------------------------------------------- > 1. More Heresy (Pete) > From: Pete Date: Sun, 18 Jan 2004 08:26:11 -0500 > Subject: More Heresy > > > Steve suggests a two (or more, by extension) tierd test > system, and I just can't get on board with this one. In the > first place, I know people who've worked real hard (I'm NOT > one of 'em) and scored in the 80s on the first pop - why > force these folks to repeat the process? ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** Subject: Digest for the period 1/20/2004 - 1/21/2004 Date: Wed, 21 Jan 2004 01:00:27 -0500 Table of contents ---------------------------------------------------------------------- 1. Exam scoring (Jeremy Bergsman) 2. Exam guide (Gordon Strong) 3. Re: Exam heresy (BillPierce`at`aol.com) 4. Re:More Heresy (Pete) (Steve Casselman) ---------------------------------------------------------------------- From: Jeremy Bergsman Date: Tue, 20 Jan 2004 02:07:54 -0500 Subject: Exam scoring I thought I would throw in a few comments on exam scoring, since these things always seem like a big secret until you've done it. It's a lot like beer scoring. How many points off do you take for an IPA that isn't quite as bitter as you'd like? How do you reward that plambic that just nailed the whole barnyard/estery thing? These are essay questions and scoresheets--there's no easy way to decide in advance how to mark an answer. When you get your exam score you get several pages back, the first page has the score (broken down into several categories), some suggestions for further study, and a section on "interpreting your score." The main part reads like this: =============================================================================== <60: Poor knowledge of brewing and/or styles with insufficient communication skills to be a judge. Generally has weak tasting skills that need development. 60s: A basic grasp of fundamentals. May have some big knowledge gaps, but has the minimum acceptable communication and judging skills. 70s: Knows basis well enough to reach the certified level, but there are some errors and gaps in the answers. Depth in answers is not necessary. 80s: Good knowledge of all subjects, with minor errors but no significant gaps. Some depth, and the essay and tasting portion should show similar ability. 90s: Excellent knowledge level with no significant errors or gaps. Good depth to answers with evidence of independent thought. Excellent tasting and communication skills. =============================================================================== Just like a new judge has to learn what makes for a "very good" beer, a new exam grader has to learn what makes a national judge. Of course we should already have a pretty good idea about this. BTW, as was said previously, on my exams I am writing about a full page of 10 point single spaced replies in addition to the first page described above. I would like to relate a personal anecdote about exam grading. It's very late and I will probably regret writing this, but.... In large part I agreed to become an exam grader because I had the perception that grading was too hard and good judges were getting low scores onthe exam. Now that I have done it for a few years I'm not so sure. I'm sure that test anxiety prevents some people from realizing their potential, but I am amazed at many of the exams I grade. They just aren't that good. I generally feel like I'm bending over backwards, giving the benefit of the doubt wherever I can. As I'm sure most of you know, on most of the questions you have to describe one or more beer styles. Believe me, I'm not marking people down for getting the OG off by 3 points. MOST people fail to describe one of the major aspects of a beer (like hop character) for MOST of the beers. A good fraction of the examinees is unable to describe at least one of the beer styles AT ALL. You wouldn't believe how few of your fellow judges know the difference between Berliner Weiss and Bavarian Weizen. We're not asking that you memorize all the numbers, but you should be familiar with all the styles! Whole parts of the answers are often left out, like the required commercial examples, or when it asks to compare the beers--they are not compared--just described. This is all probably going to make people who feel they should have gotten a 90 even more upset, but all I can say (since I've never seen an exam that could be construed by anyone to be a Master level exam) is if you know the styles and you can sit down and practice writing answers that just methodically cover all the aspects of a beer (like you are judging it: aroma, color...) and actually answer the questions, you are basically in National territory. The only thing you should have to memorize IMO is the commercial examples. If you read the examples of good answers I think you can see they are achievable. I was unhappy with my exam score. Looking back at it I have a feeling my exam wasn't as good as I thought when I turned it in. Nobody likes working that hard and not doing well. If I didn't know it would be a logistical nightmare to return the exams I would say it would be good, because I think some of the complaints would go away when people could review their exams in the cold light of day. None of this is directed at anyone who posted here, since I of course have no idea whether I've ever seen your exam, but this is my experience in the program. -- Jeremy Bergsman jeremy`at`bergsman.org http://www.bergsman.org/jeremy ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: Gordon Strong Date: Tue, 20 Jan 2004 08:39:14 -0500 Subject: Exam guide I'm also reading this thread with interest. I'd like to add a few points and provide some of the requested information. First, I do agree with some of the sentiments that it would be helpful if there were standardized deductions for common errors or at least more common examples of the level of detail required to obtain full credit (or varying levels of partial credit). Perhaps this is something the BJCP will take up in the coming year. As it is, the burden falls on the graders (and the reviewers) to assess the question in its entirety and make a holistic estimate. While this might give some variability on individual answers, when looking at the entire test the final level does tend to be more closely evaluated as to the ranking it grants. When a final score is borderline between levels, the reviewers pay careful attention to the full test to properly place it in the most appropriate rank. This point may be obvious but grading essay exams is inherently subjective, and the process of testing is inherently critical. Tests are meant as evaluation and diagnostic tools. The process of taking a test might be humbling, but I contend that it's a reasonable way of determining knowledge. There might be more accurate ways of determining knowledge, but it's hard to do better given the limitations of finding suitable exam administrators, proctors and graders. I agree with Pete Garofalo that it becomes apparent when grading exams when a person really 'gets it'. And I also agree that the quality of score sheets I get back from competitions does tend to reflect the level of the judge. For those asking about more detail on how the exam is scored and how to do well on the exam, I suggest you take a look at the presentation I gave at last year's AHA Conference in Chicago. Check out http://207.89.201.107/Presentations/Mastering%20the%20BJCP%20Exam.pdf. It does include a real-world example of the difference between an answer given a 7/10 vs. a 10/10. It also contains my lessons learned for doing well on the exam, including study, preparation, and test-taking skills. From the feedback I got, it was well-received although others who used the material might want to offer their own opinions. I've graded over 250 exams since 1997, administered or proctored 7 exams, taught 5 judge prep classes, and been an associate exam director for the last 2 years. I've judged in about 100 competitions, and entered at least that many. I have observed judges throughout the entire process and have had many discussions with judges on the pros and cons of our system. I don't think it's perfect, but it's also not as bad as some are making it seem. As you can see from the replies, the people involved with exams and exam policy at the BJCP do follow this list and do pay attention to the discussions. The process of highlighting deficiencies and offering suggestions for improvement is welcome (although the tone of the critique also does matter). Regards, Gordon Strong ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: BillPierce`at`aol.com Date: Tue, 20 Jan 2004 10:32:31 EST Subject: Re: Exam heresy I have been noting with interest the recent debate about the exam structure and grading. I believe it's healthy to discuss these issues because it promotes a clearer understanding of the expectations of the BJCP and encourages objectivity and fairness as well as accurately reflects the skills necessary to evaluate and compare beers and provide useful information to brewers. I agree that the rote memorization of the specific numbers for each style and substyle is of somewhat limited benefit. Isn't it sufficient to be able to state that a Special or Best Bitter, for example, is a low to moderate gravity English ale of medium gold to copper color, with bittering ranging from moderate to somewhat pronounced, and balanced by malt flavor, often with a little caramel sweetness in the finish? Does being able to recite the numbers (O.G. 1.039-1.045, F.G. 1.009-1.014, IBUs 20-45, SRM 6-14, ABV 3.7-4.8%) indicate that the exam taker really knows any more about the style or merely that he/she is good at parroting statistics? At every good competition (the great majority I have judged) either the organizers provide a copy of the style guidelines at each table, or at least one of the judges at each table brings a copy with him/her. Prior to the evaluation of the calibration beer, the judges read aloud the and briefly discuss the guidelines for the style, often mentioning related styles and those with which it may be compared and overlap. At that time the specific numbers and ranges are stated. Several months ago I retook the exam (I'm currently Certified) in hopes of improving my score. My knowledge of brewing and styles has certainly improved since my first exam in 1999. I have since attended the Siebel Institute, I spent some time brewing professionally and I moderate an online discussion forum (the HBD's Brews & Views) that answers a lot of homebrewing questions. I'm not encouraged when I hear that Mike Dixon, for example, did not achieve Master level on his retaking of the exam. I don't mean to single Mike out, but I know of his numerous posts to the rec.crafts.brewing newsgroup for many years; his knowledge of homebrewing is of the highest caliber in my opinion. Now I realize that brewing knowledge is only part of the exam. Highly specific knowledge of beer and the style guidelines is also required. But my point is that I am extremely confident of his ability to evaluate beer and provide detailed information about it. I know that he understands what is necessary to brew high quality beer and how to improve it. I have heard various pieces of information over the years about the range of exam scores. I have no way to verify them, so I am mentioning them here. Among these, for example, are that 40 percent of exam takers fail (that is, score below 60) and that only 4 percent score 90 or above. Another "rumor" is that there are a much greater number of scores at the top of each level (for example, 67-69 and 77-79) than at the bottom (e.g., 60-62 and 70-72). The inference is that the judges like to offer a carrot to encourage retaking of the exam, in order to suggest that with only a small amount of study it would be possible to raise the score to the next level. This is analogous to what teachers call the "plus syndrome," that is, there are often a greater number of B+ grades given than B- grades. This provides a pat on the back to students without any substantive benefit, as the grade is rounded to the letter anyway. I would very much like to see a published distribution of the scores, to let exam takers know how the scores fit over a range. I also agree that it would be very beneficial to publish actual specific examples of answers that have been graded and given a particular score. I would like to know what is necessary to receive a score of 3, 5, 7, 9 and 10, for example. This would be of great use both to prospective exam takers and graders. Additionally I will mention that the information about the published times for exam scoring and return needs to reflect the actual performance of the scorers. It has been four months now (nearly 18 weeks) since I took the exam in September, yet I still have not received my score. Let's continue this discussion. I believe it's very important to maintain the image of the BJCP as a standards organization that reflects what is truly necessary to achieve its stated goals. -- Bill Pierce Cellar Door Homebrewery Now located in Burlington, Ontario ---------------------------------------------------------------------- From: Steve Casselman Date: Tue, 20 Jan 2004 11:37:28 -0800 Subject: Re:More Heresy (Pete) Of course you would have your choice of what test to take. Are you really concerned about snobbiness over which test you took? Steve > ---------------------------------------------------------------------- > 1. More Heresy (Pete) > From: Pete Date: Sun, 18 Jan 2004 08:26:11 -0500 > Subject: More Heresy > > > Steve suggests a two (or more, by extension) tierd test > system, and I just can't get on board with this one. In the > first place, I know people who've worked real hard (I'm NOT > one of 'em) and scored in the 80s on the first pop - why > force these folks to repeat the process? ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * **********************************************************************