School Bias Among Top 25 Judges

Introduction

Last month, we provided survey data on the best judges in LD. Now we have some data on how these judges are voting. Much has been said about side bias, but what about school bias among members of the Top 25? The conclusions we can draw from this article are limited, but they raise questions about why a certain judge might be voting in odd ways: Does one school pursue a style this judge really likes? Does this judge vote less for schools in his or her region? Or is this judge just plain biased? Based on the data, these questions are unanswerable. But it’s still fun stuff to look at anyway.

Method

We gathered data from tabroom.com on the Top 25 Judges. We included all rounds at Varsity LD tournaments during the regular season, including all round robins. In looking at a particular judging record, we only considered schools that the judge had seen at least four times. After selecting down to only these rounds, we compared the percentage a given school wins in front of that judge with the percentage a given school wins in front of the pool of 25 judges.

Disclaimer

As will be discussed in the conclusion, the data displayed does not demonstrate establish any bias that can be called statistically significant. It may still be helpful to start noticing trends in case future research on judge biases is needed.

Judges might use this information to think about their own habits. Students and coaches might use it strategically when evaluating prefs.

Data

Apps. = Appearances in front of the judge

Wins = Wins in front of the judge

T-Apps., T-Wins and T-Win % = Appearances, wins and win percentage in front of the entire pool of Top 25 Judges (provided they’d judged them four times)

#1 Emily Massey

#2 Bob Overing

#3 Jonathan Wei

#4 Ryan Fink

#5 Benjamin Koh

#6 Tom Evnen

#7 Chris Theis

#8 Travis Fife

#9 Fred Ditzian

#9 Rory Jacobson

#9 Erik Legried

#12 Chris Castillo

#12 Steve Knell

#12 Adam Torson

#15 Arun Sharma

#16 Mathew Pregasen

#17 Mark Gorthey

#17 Jordan Lamothe

#17 Liz Letak

#17 Scott Wheeler

#21 Michael Harris

#21 Martin Sigalow

#23 Salim Damerdji

#23 Terrence Lonam

#23 Millman

Potential Errors

It is very possible that there are some errors in the data. Though a lot of Excel formula were used, much data was counted and manipulated by hand. We apologize for any such errors. Some other potential sources of error we encountered include rounds mysteriously missing from this year’s Harvard results that were reconstructed and independent debaters changing school codes multiple times.

Limitations and Conclusion

This is a preliminary statistical analysis of judge bias in LD debate. While it does not present any statistically significant conclusions about judge bias, it does present a large data set and reason to believe that this issue merits further investigation. A judge’s record for a particular school may differ from the norm among other top judges, but that does not automatically imply the judge is biased. And vice versa. There could be a number of confounding variables. For instance, he or she may have simply judged the best or worst debaters from a particular school. Or maybe the school is better at adapting to that judge’s paradigmatic preferences.

The data set was large but could be far more comprehensive. For one, all the data was from tabroom.com, which excludes a number of bid tournaments that did not use Tabroom software. Second, Tabroom has really only been used since Fall 2012. Some judges had hundreds of rounds logged, but others only had a few dozen.

The baseline of four rounds per school is probably still too low. Four rounds is not enough to establish a consistent trend for that judge and that school, and four was a fairly arbitrary choice We chose four because requiring a larger sample would have excluded most of the data for a few of the judges, so we erred on the side of more not less.

A more robust statistical analysis of judge bias in LD would require a logit regression model to predict how the expected outcome of a particular debate changes based on who judges said debate, examining such additional variables as tournament, side, geographic region, record disparity between the two debaters, etc. Such an analysis would require a more comprehensive data set and an immense number of person-hours.

What do you think? What does the data show? Should there be more analyses like these? What are the important variables that we should be talking about?

Thank you to Nicole VanderMeer for her help with data collection and to Tom Placido and Adam Bistagne for their input.

.

.

.

Bob co-directs Premier Debate, coaches his alma mater Loyola High School and debates on the NDT/CEDA circuit for the USC Trojan Debate Squad. His students earned 32 TOC bids in his first two years coaching. At USC, Bob qualified to the NDT and cleared at CEDA. At Loyola, Bob earned 11 bids and was a TOC finalist.