Subject: Digest for the period 4/13/2004 - 4/14/2004 Date: Wed, 14 Apr 2004 01:03:43 -0400 Table of contents ---------------------------------------------------------------------- 1. re: Telexhibitionists, imbecells, cellphoniots, et cetera (Bill Wible) 2. variance of a judge's score... (BigWayne19) 3. Quantifying Judges (Valerie Oliver) ---------------------------------------------------------------------- From: Bill Wible Date: Tue, 13 Apr 2004 12:05:47 -0400 Subject: re: Telexhibitionists, imbecells, cellphoniots, et cetera >perhaps organisers need to tell judges that mobile phones >will not be permitted in the judging rooms and that anyone >carrying one needs to go out and lock it up. Not gonna happen. We all have lives outside of judging, so get used to it. In my case, for example, I own a homebrew shop. When I judge, its just about always on a Saturday, because that seems to be the only day these things can ever be held. I've asked many, many times if these competitions could be held on Sunday. But most other people seem to think that's a bad idea. So if I'm judging on a Saturday, my store is open, and if I'm judging, than I have been able to get someone else to cover for me. They will need to get me if they have questions, and I guarantee they will. Sorry, I have no choice. When I take calls, I do apologize to the other judges, I try to keep them short as possible, I don't talk in a booming voice right where the other guys are judging, in fact, I walk away from the table and leave the room. But I have absolutely no choice. My only other option is not to judge at all. If that's what the BJCP wants, then alot of us will probably go that way. So don't start. Bill ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: BigWayne19 Date: Tue, 13 Apr 2004 11:18:16 -0700 Subject: variance of a judge's score... ---------------------------------------------------------------------- From: Steve Casselman Date: Mon, 12 Apr 2004 11:39:10 -0700 Subject: Judging the judges ...I suggest that we look at the variance of a judges score. ... ------------ however, there's the converse, too: people who score every beer 29/30/31... Suppose you judge with someone and for every beer he/she is always above or below you by a large amount ... ---------- if they're always above you (or below you) at least they're consistent , but when your or their results are random, who's correct ? .. . . you might think "hey it is just me and them." But what if you were to find out that this other judge was _never_ close to the other judges ... ------------ and the guy who only scores beers at the median is never far away from you. they never stick their neck out and have to rationalize their stance . . . this, too, has a name : central tendency. nobody wants to score a beer at 40 and nobody wants to score a beer (very) low . . . on the other hand, reality is that most beers are average (duh...) . . . ...(a) judge has a large variance in their scoring. And it is easy to find. Since the scores are put in a computer and that data is available then we have a way of finding the people that judge contrary to the community. ... ----------- if you're going to this much trouble, why not just have a suggestion box where you can suggest that your fellow judge didn't know his octanoic acid from his diacetyl ? for determining placings and best-of-show, etc., why not inflate (or deflate) a judge's tasting scores by his score on the tasting portion of the bjcp exam ? . . . ...If I get a 20 on a beer that really good I don't care what the person had to say. ----------- and yet i would want to know why / what this guy found that knocked my score down so low . . . Big ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: Valerie Oliver Date: Tue, 13 Apr 2004 14:25:28 -0700 Subject: Quantifying Judges I haven't been judging nearly as long as Steve Casselman and I disagree with trying to highlight 'problem' judges based on variance in their scoring. This could be an issue, but I don't feel it is going to be the best indicator of poor feedback. It would be interesting to see if there was a trend in judge score variance, but at most competitions that I have been at, the organizer has announced that score variance has to be within a certain spread, so any wide disagreements are hashed out at the judging table and don't show on the judging sheet. I've judged with others that are often so-many points higher/lower than me and it gives me a sense that we are being consistent in our scoring. When we encountered a larger variance or a flip-flop of who is higher/lower, then we have more to discuss. (I have known judges that have been very close in scoring compared to others, but still put down only two word comments!) The scores for a beer are important. They determine if a beer advances to the next round, but this is for a relatively small percentage of entries. Other than for bragging rights, scores don't mean much to most brewers I know. The written feedback is where most brewers, particularly less advanced brewers, learn how to improve their beer. And, more importantly, learn how to associate what they taste/smell/see in their beer with what a 'trained' person has described as what they sensed in that same beer. That teaches them how to better evaluate their own beer and thus be a better brewer. This is what I think most brewers are paying their entry fee for. I'm sure that most of us who regularly enter competitions have had an entry (or several) score significantly different at different competitions. Even when the competitions were relatively close together in time and space. The differences in scores could be caused by bottling/handling issues, but I don't put much stock in scores. I look at the comments to improve my brewing and evaluation skills. That is were I think we, the BJCP, should spend our time and energy to educate and improve judges. The scoring will naturally improve as a result of this effort. Exactly what that effort will be...I'm still scratching my head on that one, but as has been hinted at in this forum, Continuing Education opportunities and/or requirements for judges seems to be a good place to start. Valerie Oliver Seattle area ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** Subject: Digest for the period 4/13/2004 - 4/14/2004 Date: Wed, 14 Apr 2004 01:03:43 -0400 Table of contents ---------------------------------------------------------------------- 1. re: Telexhibitionists, imbecells, cellphoniots, et cetera (Bill Wible) 2. variance of a judge's score... (BigWayne19) 3. Quantifying Judges (Valerie Oliver) ---------------------------------------------------------------------- From: Bill Wible Date: Tue, 13 Apr 2004 12:05:47 -0400 Subject: re: Telexhibitionists, imbecells, cellphoniots, et cetera >perhaps organisers need to tell judges that mobile phones >will not be permitted in the judging rooms and that anyone >carrying one needs to go out and lock it up. Not gonna happen. We all have lives outside of judging, so get used to it. In my case, for example, I own a homebrew shop. When I judge, its just about always on a Saturday, because that seems to be the only day these things can ever be held. I've asked many, many times if these competitions could be held on Sunday. But most other people seem to think that's a bad idea. So if I'm judging on a Saturday, my store is open, and if I'm judging, than I have been able to get someone else to cover for me. They will need to get me if they have questions, and I guarantee they will. Sorry, I have no choice. When I take calls, I do apologize to the other judges, I try to keep them short as possible, I don't talk in a booming voice right where the other guys are judging, in fact, I walk away from the table and leave the room. But I have absolutely no choice. My only other option is not to judge at all. If that's what the BJCP wants, then alot of us will probably go that way. So don't start. Bill ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: BigWayne19 Date: Tue, 13 Apr 2004 11:18:16 -0700 Subject: variance of a judge's score... ---------------------------------------------------------------------- From: Steve Casselman Date: Mon, 12 Apr 2004 11:39:10 -0700 Subject: Judging the judges ...I suggest that we look at the variance of a judges score. ... ------------ however, there's the converse, too: people who score every beer 29/30/31... Suppose you judge with someone and for every beer he/she is always above or below you by a large amount ... ---------- if they're always above you (or below you) at least they're consistent , but when your or their results are random, who's correct ? .. . . you might think "hey it is just me and them." But what if you were to find out that this other judge was _never_ close to the other judges ... ------------ and the guy who only scores beers at the median is never far away from you. they never stick their neck out and have to rationalize their stance . . . this, too, has a name : central tendency. nobody wants to score a beer at 40 and nobody wants to score a beer (very) low . . . on the other hand, reality is that most beers are average (duh...) . . . ...(a) judge has a large variance in their scoring. And it is easy to find. Since the scores are put in a computer and that data is available then we have a way of finding the people that judge contrary to the community. ... ----------- if you're going to this much trouble, why not just have a suggestion box where you can suggest that your fellow judge didn't know his octanoic acid from his diacetyl ? for determining placings and best-of-show, etc., why not inflate (or deflate) a judge's tasting scores by his score on the tasting portion of the bjcp exam ? . . . ...If I get a 20 on a beer that really good I don't care what the person had to say. ----------- and yet i would want to know why / what this guy found that knocked my score down so low . . . Big ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * ********************************************************************** ---------------------------------------------------------------------- From: Valerie Oliver Date: Tue, 13 Apr 2004 14:25:28 -0700 Subject: Quantifying Judges I haven't been judging nearly as long as Steve Casselman and I disagree with trying to highlight 'problem' judges based on variance in their scoring. This could be an issue, but I don't feel it is going to be the best indicator of poor feedback. It would be interesting to see if there was a trend in judge score variance, but at most competitions that I have been at, the organizer has announced that score variance has to be within a certain spread, so any wide disagreements are hashed out at the judging table and don't show on the judging sheet. I've judged with others that are often so-many points higher/lower than me and it gives me a sense that we are being consistent in our scoring. When we encountered a larger variance or a flip-flop of who is higher/lower, then we have more to discuss. (I have known judges that have been very close in scoring compared to others, but still put down only two word comments!) The scores for a beer are important. They determine if a beer advances to the next round, but this is for a relatively small percentage of entries. Other than for bragging rights, scores don't mean much to most brewers I know. The written feedback is where most brewers, particularly less advanced brewers, learn how to improve their beer. And, more importantly, learn how to associate what they taste/smell/see in their beer with what a 'trained' person has described as what they sensed in that same beer. That teaches them how to better evaluate their own beer and thus be a better brewer. This is what I think most brewers are paying their entry fee for. I'm sure that most of us who regularly enter competitions have had an entry (or several) score significantly different at different competitions. Even when the competitions were relatively close together in time and space. The differences in scores could be caused by bottling/handling issues, but I don't put much stock in scores. I look at the comments to improve my brewing and evaluation skills. That is were I think we, the BJCP, should spend our time and energy to educate and improve judges. The scoring will naturally improve as a result of this effort. Exactly what that effort will be...I'm still scratching my head on that one, but as has been hinted at in this forum, Continuing Education opportunities and/or requirements for judges seems to be a good place to start. Valerie Oliver Seattle area ********************************************************************** * JudgeNet - the beer judge digest * * Send plain text only, no HTML, MIME, encoded text or attachments * * Send subscription requests & changes to judge-request`at`synchro.com * **********************************************************************