Q: What are the disadvantages of using balloting as a selection methodology?
A: The balloting process is inconsistent with the principles and methods of science.
There are 8 major disadvantages to using balloting:

(1) the inherent subjectivity of balloting systems
Balloting is opinion-based rather than fact or performance-based.

(2) the arbitrariness of scoring in balloting systems
On the one hand, an award might count second and third place votes in addition to first place votes. On the other hand, a separate award might only count first place votes. The decision on whether or not to include second and third place votes is arbitrary, and so is the decision on how to score them.

(3) the external invalidity of balloting systems
If there are discrepancies in voting behaviors of national selection committee panelists in different regions of the country, then an inability to reproduce results with different participants is likely. In other words, a system is externally invalid when disagreement results from geographic distribution rather than level of expertise or quality of observation.*

(4) the internal invalidity of balloting systems
A system is internally invalid when nebulous terms like “most outstanding player” are not operationally defined. This leads to divergent conceptual interpretations amongst voters, who may interpret it to mean any of the following: the best player, the best player on the best team, the player with the most NFL potential, the player with the most media attention, the most valuable player, the highest performing player on a team, the highest performing player overall, etc..

(5) the form of balloting
Plurality voting arrangements, for instance, systematically increase:
(a) tactical voting,
(b) the importance of media attention,
(c) voting against candidates rather than voting for candidates of true preference,
(d) bias against new players,
(e) wasted votes, and
(f) the likelihood of a non-majority candidate winning (Duverger’s Law).

(6) a lack of transparency
If a selection process is kept fully confidential, the actual ballot results, national selection committee membership, number of votes placed, percentage of votes placed, and number of votes wasted never become public knowledge.

(7) non-representative sampling of voting panelists
For instance, if the membership of a particular national selection committee does not reflect the geographic diversity of the U.S. proportionally, then geographic bias is likely.

(8) pre-emptive candidate eliminations
If voters are forced to make semifinalist and finalist cuts during the season, then players with impressive performances after these cuts are likely to be excluded.

*In a 2009 meta-analysis of conference individual postseason awards, coach balloting and media balloting agreed only 55.7% of the time (n=307). Note: national balloting-based awards necessarily increase both the size of candidate samples and the geographic distribution of voters, and are, therefore, more likely to yield lower external validity factors.