Subject: Digest for the period 4/6/2004 - 4/7/2004 Date: Wed, 07 Apr 2004 01:01:11 -0400 Table of contents ---------------------------------------------------------------------- 1. Master routes and the BJCP Exam (Michael L. Hall) 2. Score Sheet Quality (gornicwm`at`earthlink.net) 3. Re: Test Option (Steve Casselman) 4. two-track testing (Bob Paolino) ---------------------------------------------------------------------- From: Michael L. Hall Date: Tue, 6 Apr 2004 01:23:16 -0600 Subject: Master routes and the BJCP Exam Greetings, First off, I would like to thank everyone involved for the insightful discussion of the "alternate path to Master" thread. [1] In spite of the strong emotions on both sides of this issue, the comments have mostly stayed to the heart of the matter. The discussion has strayed to the topic of the BJCP Exam itself. I have thought about this topic a bit in the past, so I would like to share my thoughts. However, note that these are my thoughts only, and are not a pronouncement or endorsement by the BJCP (I'm currently the president, in case you didn't remember). I haven't even shared them with the BJCP Board yet (at least I don't think I have). At any rate, this is just me speaking. ===== IMO, the BJCP should be about helping people become good judges of beer. It should help them learn about styles (taste, aroma, appearance, ingredients and history, etc.); it should help them develop their palates; and it should help them to be more expressive as they convey their opinions to the brewers of the beers they are reviewing. On another front, the BJCP exam is seen by some to be an onerous exercise in writing, and is seen by others to be capriciously graded. While I most vehemently disagree with the latter -- I know our graders are working very hard to make the exam grading as objective (ahem) as they can -- I have to admit to be in the camp of those who hate the writing. As those that are on the board (or the kind readers of this posting) will attest, I can be very long-winded at times when behind a keyboard, but my writing suffers when forced to write linearly (instead of skipping about) with a pen and paper. My personal solution to the two problems in the previous two paragraphs (i.e. BJCP is about teaching people to judge and the current test is onerous) is as follows. Again, I must emphasize that this is me speaking, not the BJCP. Each of the ten questions on the current exam is meant to test some set of knowledge. I don't think it would be hard for the graders to write down the minimal set of "facts" that are associated with any particular question. I'm guessing this might be twenty or thirty facts per essay question. What if we came up with a set of objective questions [2] to use for the test? We could certainly make a test that would be "as hard as" the current test. We could test the exact same knowledge that we're testing now, especially if we explicitly wrote down all the facts required for the solution to a particular essay question. BUT DON'T STOP READING YET... the new idea is yet to come... I know, an objective test has been suggested before. Here's the new thing: I suggest that we get the pool of questions by polling our membership. I imagine a web site with an interface which allows you to enter new questions and answers. I imagine a web site that allows you to practise the test on-line by answering questions and having your answers judged in real-time. I imagine a web site that keeps track of your cumulative score as you continue answering questions. And I imagine a web site that allows you to "argue" with the answers by posting a reply (preferably, with back-up references) as to why your answer is really right. Imagine that -- any beer-swiller could practise the test on-line! Just go to the BJCP.org site and click on "I want to test my Beer IQ!" and you would be taken to a page which asks something like Question #387: What chemical ion is associated with the "Burton Snatch"? a. Carbonate b. Sulfate c. Chloride d. Phosphate He could choose an answer, click a button, and then would be taken to the currently-accepted BJCP answer (b. Sulfate), along with a write-up with a few references. (I've actually been looking at this lately, so that's why I asked this question.) The question number could be used to come back to the question later if you had a comment. But what if our hypothetical beer-swiller disagreed with the answer? Then he could check the box saying "I disagree" and enter some explanatory text. This would get reviewed by a grader (a more productive use of grader time, IMO) and the question might be changed. There would also be a spot to enter completely new questions. And questions would be a mix of hard (almost impossible?) questions and easy questions to encourage beginners. Maybe there would even be a graph of the percentage of people that have gotten a particular question right. The actual BJCP exam would then be questions taken straight from the on-line test. What are we worried about anyway, that people might memorize the answers? Good, let them memorize the answers! That's what we want! I'm thinking that something around 200-300 questions in three hours would be about as hard as our current test. So, to summarize, here are what I see as the benefits: - Easier to score -- which means less volunteer time and quicker results. - Allows newbies or even people not interested in taking the exam to test their knowledge. - Allows you to do a real practise on your own -- when you get to a level you're satisfied with, take the written test. - Leverages the abilities and time of our knowledgeable membership. - Helps people learn about beer styles, which IMO is one of the missions of the BJCP. As for the tasting part of the test, I am *not* suggesting that we change it or the weighting (30%). I only suggest a change for the essay part. ===== In closing, I want to caution once again that this idea is mine and not the BJCP's. As president, I would of course want to get the opinions of our very able graders, exam directors and membership, and approval by the board before a change that was this radical was introduced. I just wanted to share this idea with you to hear your opinions about it. I've been thinking about this for a while and I was excited by this solution. Please don't rip it to shreds. :-) Good Beer to You, Mike [1] - Point well taken, Bob (Paolino), about the inherent bias in this language -- but that's how we started, so I'll continue to use it here. [2] - By objective questions, I mean multiple choice, true/false, and to a lesser degree, short answer. I only have reservations against short answer questions because it's easy for there to be multiple answers that the test givers haven't thought of that are still valid answers (or are close, but incomplete). I have served as a Science Judge at our state Science Bowl for almost ten years now, and there is always a pivotal short-answer question that I have to rule on on-the-fly -- that's the toughest part of the job. ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: gornicwm`at`earthlink.net Date: Tue, 6 Apr 2004 09:33:52 -0400 (GMT-04:00) Subject: Score Sheet Quality Fellow Judges, I simply wanted to post a friendly, humble reminder of our "duty" to "properly" fill out the scoresheets of the upcoming AHA Regionals as completely as possible. We should also attempt to be insightful with the comments that we record about a particular brew. I know that I am preaching to the choir with most judges - I have worked with some OUTSTANDING judges. However, as a judge, it is easy...quite easy in some respects...to spot the lack of quality that some judges put into the scoring of a beer. Long story short. I reviewed some of my past scoresheets (last years AHA Regional) and observed that many of the judges scored these beers with one word comments or no comments at all. I paid $8 for THIS!!! These judges were also ranked as Certified or National status. I have also judged along side a few of these individuals and noticed that I am still feverishly writing as they kicking back and relaxing. Fast writers? I don't think so. What is another judge to do in this respect? I mean, if you notice that another judge is doing a poor job of filling out a scoresheet, do you call them on it? Not a real good way to make friends, I would think. Is there a way of policing the the quality that I do not know about? My Humble Request: Judges, maintain a sense of pride and professionalism while judging beer. Our rank, coupled with the quality that we display while scoring a beer (comments, perception, insights...etc) is reflective of the entire BJCP community. Being that we ARE the BJCP we all look bad -- keep that in mind. I am proud to belong to the BJCP, but I find it hard to explain our passion and knowledge of beer while looking at a lame excuse of a scoresheet from a fellow ranking member. Sincerely, Bill Gornicki ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: Steve Casselman Date: Tue, 6 Apr 2004 07:49:07 -0700 Subject: Re: Test Option I like match type questions better than multiple choice . Match the beer styles to their country of origin. A Pale Ale 1 Germany B Lager 2 Belgium C Pilsner 3 England D Lambic 4 Checkoslovakia You could have 2 or 3 of these type questions Off flavors: Identify the source of each off flavor A Cardboard B Bandaid C Burnt rubber D Meaty Anwsers would be A Oxidation B Phenols C Phenols D Autolysis In the styles below indicate whether the beers are balanced toward more hoppy or more malty in their flavor profile Pale Ale American Lager Lambic (haha trick question!!!) Scottish Ale Porter Bock I would have them judge 2 beers one good and one flawed. Also I think you would still have one compare and contrast question. Steve ---------------------------------------------------------------------- From: Bob Paolino Date: Tue, 6 Apr 2004 10:50:54 -0400 Subject: two-track testing David Craft wrote on Sun, 4 Apr 2004 20:15:52 -0400: > Greetings, > > Since the topic of two tests keeps popping up, one for newer folks and a > more advanced test for Masters or higher, let's move this forward. National, not Master. I think the suggestions have been for a Certified ceiling on the optional "beginner" (for lack of better word) test and keeping the existing test for eligibility for the full range of rankings with the existing scoring (or possibly a new advancement grid, described by its detractors as the "alternate path"). What hasn't been addressed about this new option being floated is how it would be scored. The current full-range exam has a 60 passing grade, with advancement (with the required experience) at 70, 80, and 90. But a 70 on the proposed "beginner" exam wouldn't have the same meaning as a 70 on the current, more advanced one. I don't see a lot of sense in saying (for this entry level test) that you still get Recognised at 60 and then Certified for the entire range 70 and above. After all, wouldn't that "cheapen" the Recognised rank (grin)? Maybe with a new test you'd need a 75 to pass and an 85 or 90 to qualify for Certified. If you're going to offer an easier test as an option, it seems obvious that you'd have to score higher than you would on the more difficult one (and/or require more experience) to achieve the same rank. > How about posting multiple choice questions that could be used for first > time test takers? > Seeing how that goes, when can see if that is viable. Writing good multiple choice questions is tough. I wrote a book full of them in graduate school when I did the instructors' manual (complete with multiple choice questions for the use of professors/instructors who don't give proper written exams) for my advisor's undergraduate textbook. It wasn't exactly something you just crank out in a week. If you were to look in the JudgeNet archives, you'd find extended discussion about the possibility of changing the BJCP exam (THE exam, not an optional entry level exam) to multiple choice. I can see that this option could be useful for getting lots of exams graded quickly and for potential judges who simply want the certification to judge but have little interest in judging enough to move to higher ranks. For those who want to move higher, however, it guarantees the need for a retake (not a lot different from the current situation of a new homebrewer/taster taking the exam early in the hobby and passing but not scoring exceptionally well, so requiring a retake later to advance). On the whole, however, I'm not convinced that it's a good idea or necessary. A relative novice who does a decent amount of work preparing for the exam is fairly likely to pass the exam and qualify for that recognised or even certified rank. You don't need a new exam to accomplish that. The advantage I see of the current exam compared to a proposed entry-level multiple choice exam is that I think people will work a lot harder preparing for the former than for the latter, and the learning that occurs in doing that would seem to make a better beginning judge. Just my 37 cents' worth. Bob Paolino UWCU members: Please vote this month for your new board. "Homer, why don't you get one of those hands-free phones? It's the next best thing to paying attention to the road." --Barney, The Simpsons ( ) ASCII ribbon campaign X against HTML e-mail: / \ Friends don't send friends HTML-bloated messages! A: Because it reverses the logical flow of conversation. Q: Why is top posting frowned upon? ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** Subject: Digest for the period 4/6/2004 - 4/7/2004 Date: Wed, 07 Apr 2004 01:01:11 -0400 Table of contents ---------------------------------------------------------------------- 1. Master routes and the BJCP Exam (Michael L. Hall) 2. Score Sheet Quality (gornicwm`at`earthlink.net) 3. Re: Test Option (Steve Casselman) 4. two-track testing (Bob Paolino) ---------------------------------------------------------------------- From: Michael L. Hall Date: Tue, 6 Apr 2004 01:23:16 -0600 Subject: Master routes and the BJCP Exam Greetings, First off, I would like to thank everyone involved for the insightful discussion of the "alternate path to Master" thread. [1] In spite of the strong emotions on both sides of this issue, the comments have mostly stayed to the heart of the matter. The discussion has strayed to the topic of the BJCP Exam itself. I have thought about this topic a bit in the past, so I would like to share my thoughts. However, note that these are my thoughts only, and are not a pronouncement or endorsement by the BJCP (I'm currently the president, in case you didn't remember). I haven't even shared them with the BJCP Board yet (at least I don't think I have). At any rate, this is just me speaking. ===== IMO, the BJCP should be about helping people become good judges of beer. It should help them learn about styles (taste, aroma, appearance, ingredients and history, etc.); it should help them develop their palates; and it should help them to be more expressive as they convey their opinions to the brewers of the beers they are reviewing. On another front, the BJCP exam is seen by some to be an onerous exercise in writing, and is seen by others to be capriciously graded. While I most vehemently disagree with the latter -- I know our graders are working very hard to make the exam grading as objective (ahem) as they can -- I have to admit to be in the camp of those who hate the writing. As those that are on the board (or the kind readers of this posting) will attest, I can be very long-winded at times when behind a keyboard, but my writing suffers when forced to write linearly (instead of skipping about) with a pen and paper. My personal solution to the two problems in the previous two paragraphs (i.e. BJCP is about teaching people to judge and the current test is onerous) is as follows. Again, I must emphasize that this is me speaking, not the BJCP. Each of the ten questions on the current exam is meant to test some set of knowledge. I don't think it would be hard for the graders to write down the minimal set of "facts" that are associated with any particular question. I'm guessing this might be twenty or thirty facts per essay question. What if we came up with a set of objective questions [2] to use for the test? We could certainly make a test that would be "as hard as" the current test. We could test the exact same knowledge that we're testing now, especially if we explicitly wrote down all the facts required for the solution to a particular essay question. BUT DON'T STOP READING YET... the new idea is yet to come... I know, an objective test has been suggested before. Here's the new thing: I suggest that we get the pool of questions by polling our membership. I imagine a web site with an interface which allows you to enter new questions and answers. I imagine a web site that allows you to practise the test on-line by answering questions and having your answers judged in real-time. I imagine a web site that keeps track of your cumulative score as you continue answering questions. And I imagine a web site that allows you to "argue" with the answers by posting a reply (preferably, with back-up references) as to why your answer is really right. Imagine that -- any beer-swiller could practise the test on-line! Just go to the BJCP.org site and click on "I want to test my Beer IQ!" and you would be taken to a page which asks something like Question #387: What chemical ion is associated with the "Burton Snatch"? a. Carbonate b. Sulfate c. Chloride d. Phosphate He could choose an answer, click a button, and then would be taken to the currently-accepted BJCP answer (b. Sulfate), along with a write-up with a few references. (I've actually been looking at this lately, so that's why I asked this question.) The question number could be used to come back to the question later if you had a comment. But what if our hypothetical beer-swiller disagreed with the answer? Then he could check the box saying "I disagree" and enter some explanatory text. This would get reviewed by a grader (a more productive use of grader time, IMO) and the question might be changed. There would also be a spot to enter completely new questions. And questions would be a mix of hard (almost impossible?) questions and easy questions to encourage beginners. Maybe there would even be a graph of the percentage of people that have gotten a particular question right. The actual BJCP exam would then be questions taken straight from the on-line test. What are we worried about anyway, that people might memorize the answers? Good, let them memorize the answers! That's what we want! I'm thinking that something around 200-300 questions in three hours would be about as hard as our current test. So, to summarize, here are what I see as the benefits: - Easier to score -- which means less volunteer time and quicker results. - Allows newbies or even people not interested in taking the exam to test their knowledge. - Allows you to do a real practise on your own -- when you get to a level you're satisfied with, take the written test. - Leverages the abilities and time of our knowledgeable membership. - Helps people learn about beer styles, which IMO is one of the missions of the BJCP. As for the tasting part of the test, I am *not* suggesting that we change it or the weighting (30%). I only suggest a change for the essay part. ===== In closing, I want to caution once again that this idea is mine and not the BJCP's. As president, I would of course want to get the opinions of our very able graders, exam directors and membership, and approval by the board before a change that was this radical was introduced. I just wanted to share this idea with you to hear your opinions about it. I've been thinking about this for a while and I was excited by this solution. Please don't rip it to shreds. :-) Good Beer to You, Mike [1] - Point well taken, Bob (Paolino), about the inherent bias in this language -- but that's how we started, so I'll continue to use it here. [2] - By objective questions, I mean multiple choice, true/false, and to a lesser degree, short answer. I only have reservations against short answer questions because it's easy for there to be multiple answers that the test givers haven't thought of that are still valid answers (or are close, but incomplete). I have served as a Science Judge at our state Science Bowl for almost ten years now, and there is always a pivotal short-answer question that I have to rule on on-the-fly -- that's the toughest part of the job. ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: gornicwm`at`earthlink.net Date: Tue, 6 Apr 2004 09:33:52 -0400 (GMT-04:00) Subject: Score Sheet Quality Fellow Judges, I simply wanted to post a friendly, humble reminder of our "duty" to "properly" fill out the scoresheets of the upcoming AHA Regionals as completely as possible. We should also attempt to be insightful with the comments that we record about a particular brew. I know that I am preaching to the choir with most judges - I have worked with some OUTSTANDING judges. However, as a judge, it is easy...quite easy in some respects...to spot the lack of quality that some judges put into the scoring of a beer. Long story short. I reviewed some of my past scoresheets (last years AHA Regional) and observed that many of the judges scored these beers with one word comments or no comments at all. I paid $8 for THIS!!! These judges were also ranked as Certified or National status. I have also judged along side a few of these individuals and noticed that I am still feverishly writing as they kicking back and relaxing. Fast writers? I don't think so. What is another judge to do in this respect? I mean, if you notice that another judge is doing a poor job of filling out a scoresheet, do you call them on it? Not a real good way to make friends, I would think. Is there a way of policing the the quality that I do not know about? My Humble Request: Judges, maintain a sense of pride and professionalism while judging beer. Our rank, coupled with the quality that we display while scoring a beer (comments, perception, insights...etc) is reflective of the entire BJCP community. Being that we ARE the BJCP we all look bad -- keep that in mind. I am proud to belong to the BJCP, but I find it hard to explain our passion and knowledge of beer while looking at a lame excuse of a scoresheet from a fellow ranking member. Sincerely, Bill Gornicki ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: Steve Casselman Date: Tue, 6 Apr 2004 07:49:07 -0700 Subject: Re: Test Option I like match type questions better than multiple choice . Match the beer styles to their country of origin. A Pale Ale 1 Germany B Lager 2 Belgium C Pilsner 3 England D Lambic 4 Checkoslovakia You could have 2 or 3 of these type questions Off flavors: Identify the source of each off flavor A Cardboard B Bandaid C Burnt rubber D Meaty Anwsers would be A Oxidation B Phenols C Phenols D Autolysis In the styles below indicate whether the beers are balanced toward more hoppy or more malty in their flavor profile Pale Ale American Lager Lambic (haha trick question!!!) Scottish Ale Porter Bock I would have them judge 2 beers one good and one flawed. Also I think you would still have one compare and contrast question. Steve ---------------------------------------------------------------------- From: Bob Paolino Date: Tue, 6 Apr 2004 10:50:54 -0400 Subject: two-track testing David Craft wrote on Sun, 4 Apr 2004 20:15:52 -0400: > Greetings, > > Since the topic of two tests keeps popping up, one for newer folks and a > more advanced test for Masters or higher, let's move this forward. National, not Master. I think the suggestions have been for a Certified ceiling on the optional "beginner" (for lack of better word) test and keeping the existing test for eligibility for the full range of rankings with the existing scoring (or possibly a new advancement grid, described by its detractors as the "alternate path"). What hasn't been addressed about this new option being floated is how it would be scored. The current full-range exam has a 60 passing grade, with advancement (with the required experience) at 70, 80, and 90. But a 70 on the proposed "beginner" exam wouldn't have the same meaning as a 70 on the current, more advanced one. I don't see a lot of sense in saying (for this entry level test) that you still get Recognised at 60 and then Certified for the entire range 70 and above. After all, wouldn't that "cheapen" the Recognised rank (grin)? Maybe with a new test you'd need a 75 to pass and an 85 or 90 to qualify for Certified. If you're going to offer an easier test as an option, it seems obvious that you'd have to score higher than you would on the more difficult one (and/or require more experience) to achieve the same rank. > How about posting multiple choice questions that could be used for first > time test takers? > Seeing how that goes, when can see if that is viable. Writing good multiple choice questions is tough. I wrote a book full of them in graduate school when I did the instructors' manual (complete with multiple choice questions for the use of professors/instructors who don't give proper written exams) for my advisor's undergraduate textbook. It wasn't exactly something you just crank out in a week. If you were to look in the JudgeNet archives, you'd find extended discussion about the possibility of changing the BJCP exam (THE exam, not an optional entry level exam) to multiple choice. I can see that this option could be useful for getting lots of exams graded quickly and for potential judges who simply want the certification to judge but have little interest in judging enough to move to higher ranks. For those who want to move higher, however, it guarantees the need for a retake (not a lot different from the current situation of a new homebrewer/taster taking the exam early in the hobby and passing but not scoring exceptionally well, so requiring a retake later to advance). On the whole, however, I'm not convinced that it's a good idea or necessary. A relative novice who does a decent amount of work preparing for the exam is fairly likely to pass the exam and qualify for that recognised or even certified rank. You don't need a new exam to accomplish that. The advantage I see of the current exam compared to a proposed entry-level multiple choice exam is that I think people will work a lot harder preparing for the former than for the latter, and the learning that occurs in doing that would seem to make a better beginning judge. Just my 37 cents' worth. Bob Paolino UWCU members: Please vote this month for your new board. "Homer, why don't you get one of those hands-free phones? It's the next best thing to paying attention to the road." --Barney, The Simpsons ( ) ASCII ribbon campaign X against HTML e-mail: / \ Friends don't send friends HTML-bloated messages! A: Because it reverses the logical flow of conversation. Q: Why is top posting frowned upon? ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * **********************************************************************