How Good Is That Goalie?

Evaluating a goalie's true skill requires deep statistical analysis and plenty of data.

FE_PR_110504.science.hockey.jpg
+ More

Gabriel Desjardins, a statistical analyst and writer specializing in hockey said that he thought Schuckers' work was interesting. "I think he needs to take a larger sample and see how this holds over."

Desjardins contends that good defenses do not in fact impact the distribution of shots on the ice in a significant way. "[In] the NHL, the talent distribution is so small that you don't see this kind of effect except in the short run only," he said.

"[Desjardins] thinks that the difference is so small that you shouldn't bother with it and I think the difference is small but large enough that you should bother with it," Awad said. "Most people tend to be in one of those two camps."

That good defenses would force worse shots makes intuitive sense, but many things that make intuitive sense don't work out the obvious way when it comes to elite athletes.

Desjardins claimed that defenses can be best evaluated by looking at how many shots and scoring chances they allow.

Schuckers admitted that the issue of whether shots are randomly distributed or impacted by the defense is not settled.

"I think that many folks don't appreciate the difference and I don't think we as a hockey stats community have a strong enough handle on these processes to know which is the valid perspective," Schuckers said.

Icy Data

There is another problem in figuring this out: the quality of the data on which it is based. Awad said that some arenas are known to record incorrect shot distance information, while others consistently record suspiciously high or low shot totals.

And this is only part of the issue. Both Awad and Desjardins said that the variation in skill of starting goaltenders is very small.

"Most people agree that it takes a good 3 years of data for a goaltender to get a decent idea of how good he really is," said Awad.

With so many factors at play, deciding on a single goalie statistic might be impossible. What does it mean to ask who was the best goalie, home run hitter, or 3-point shooter? Is the question who performed the best or who should be the best going forward? Recording the most likely outcome is different than recording what actually happened.

"I guess that I am not interested in a goalie's skill level, but in how they perform. One year's worth of data is just that: one year's worth," Schuckers said. "Would I make a personnel decision on one year's worth of data? Probably not."

Proclaiming the best goalie of the playoffs based on 10 games is an even more questionable decision. Yet, one more solid performance from Roloson could carry his Lightning halfway through the Stanley Cup playoffs and into the Eastern Conference finals.

---

 Follow U.S. News Science on Twitter.