by Nate Silver

Most simulation league and Scoresheet owners invariably tend to make one of two mistakes when building and maintaining a bullpen.

Owners who are newer to this type of competition often have come from a rotisserie background. In roto play, saves are of critical importance, so these owners tend to overemphasize relief pitching, and in particular the importance of having a closer. An average relief pitcher will pitch only about one-third as many innings as an average starter; for that reason, you should almost never draft any reliever if a good starting pitcher is still available.

Based on how many runs they can be expected to save over the course of the season, a starter with a 3.90 ERA over 200 innings is roughly as valuable as a reliever who posts a 2.50 ERA over 75 innings.

On the other hand, many veteran owners tend to make the opposite mistake, ignoring their bullpen completely. Relief pitchers certainly won't pitch as many innings as starters, but the best couple of relievers on a team will pitch most of their innings when the game is on the line, giving them a disproportionate amount of influence over the game's outcome. It should be no surprise, then, that when Scoresheet periodically publishes its list of which players have had the greatest effect on their teams' won-loss percentage, the list always includes at least a couple of relievers.

While there is really no substitute for experience in figuring out how you can make the best use of your bullpen, following these six guidelines can help your team to avoid the same fate that Mitch Williams suffered in the '93 World Series.

Don't assume that, once you've drafted Trevor Hoffman or Mariano Rivera, your work in building a bullpen is done. It's important to have as many quality relief innings as possible.

But quality relief pitchers are hard to come by. I like to use a stat that I call the "quality season" to evaluate relievers. A pitcher who throws predominantly in relief accumulates "half" a quality season when his ERA is no greater than 3.50, and the other "half" if he pitches for 50 or more innings.

Only five relief pitchers had a "quality season" in each year from 1995-1997: John Franco, Greg McMichael, Troy Percival, John Wetteland and Mark Wohlers.

Eleven other pitchers have had 2.5 quality seasons in those three years, missing on one of the criteria in one year. They are: Rod Beck, Ricky Bottalico, Mike Fetters, Roberto Hernandez, Trevor Hoffman, Todd Jones, T.J. Mathews, Jeff Nelson, Rob Nen, Jesse Orosco, Heathcliff Slocumb and Billy Wagner. Picking up any two of these relievers would have given you a pretty good 'pen.

While Scoresheet claims that it uses saves that are accumulated in the Major League to help determine how a pitcher performs in clutch situations, this effect is so small that it may safely be ignored. While it is important to pay attention to a reliever's usage pattern (see below), you shouldn't even include a column for saves on your pre-draft worksheet.

ERA is certainly a better indicator of performance than the save, but it has its own problems. Because of the relatively small number of innings that relievers pitch, a couple of bad outings can have a dramatic and deceptive effect on an ERA for the season. Not only can ERA be a misleading predictor of future performance, it can also be a misleading indicator of past performance. Relievers can differ greatly in how well they deal with inherited runners without this phenomenon having any effect on their ERA.

The Base Performance Value (BPV) system on Baseball HQ uses strikeout rate, strikeout-to-walk ratio, and home run rate as its primary criteria for predicting pitcher performance. Focusing on those "secondary" numbers is especially important when dealing with relief pitchers. Looking at innings pitched is important too.

One peculiar aspect of Scoresheet is that playing time is divided up into separate "weeks."

Unfortunately, the weeks when you need a lot of relief pitching will not always coincide with the weeks when a player throws a lot of innings in the major leagues. What's worse, if a reliever makes only one or two appearances in the major leagues in a week, one bad outing will completely ruin any chance he has to pitch effectively for you.

It's very important, therefore, to look for relievers who are used as consistently by their major league teams as possible. This means avoiding those who have only marginal roles in their respective bullpens, and sometimes avoiding closers who are used sporadically by their managers.

One potential mistake is to try and "save" all your best relief pitchers for the last couple of innings.

More often than not, this strategy results in your good relievers running out of games to play in Scoresheet before they run out of available innings. So long as the game is not out of hand, the sixth inning is every bit as important as the ninth inning, and your "earliest inning" numbers ought to reflect this. (This is also one of the stronger arguments against listing a "closer," and a very strong argument against listing two "closers").

Hook numbers give you the opportunity to do some micro-managing, but not nearly at the depth that occurs in a real game. As a consequence, because it is hard to replicate their usage patterns by using hooks, left-handed and right-handed "specialists" are not very important in Scoresheet. This is especially true as the simulation only uses real platoon splits for batters, and not for pitchers.

With regard to questions such as to whether or not to use a closer, and what hook numbers and "earliest innings" to use, there aren't really any right or wrong answers, because these all depend so much on the composition of your team. The more quality relief pitchers you have, the lower your hooks should generally be, the earlier your earliest innings should be, and the lesser the benefit of listing a single "closer".

Depth of starting pitching matters, too. If your team needs extra relief pitching because your starters don't pitch enough innings, using low hook numbers can lead all too easily to the dreaded "Pitcher, AAA". For the same reason, even if your team has an abundance of relief pitching, a weak starting rotation should prompt you to designate one or two "mop-up" guys with very high hooks.