Try guessing how many pennies are in a large jar.
It’s a tough challenge. But as it turns out, the median of guesses from many people, with the extremes tossed out, actually provides a close estimate of the actual penny count.
Carnegie Mellon University researchers used this “wisdom of the crowds” approach to win a friendly competition sponsored by the U.S. Centers for Disease Control and Prevention in predicting week-to-week influenza activity levels in 10 regions of the nation, along with national totals.
For the 2016-2017 flu season, CMU’s two systems finished first and second among 28 submissions from other universities, government agencies and private organizations, with the “wisdom of the crowd” method achieving the highest “skill score.”
For three years now, the systems developed by the Delphi group in CMU’s School of Computer Science have predicted national influenza activity levels better than the other submissions. In the past year, CMU’s individual systems even did better than an ensemble prediction — one involving all 28 systems combined.
The field is still young with hopes of annual improvements, much the way weather forecasting has grown in sophistication over the past 150 years.
“We’re gratified that our forecasting methods continue to perform as well as they do, but it’s important to remember that epidemiological forecasting remains in its infancy,” said Roni Rosenfeld, Delphi leader and professor in the CMU School of Computer Science’s Machine Learning Department and Language Technologies Institute. “I can tell you I’m not happy with the current state of the art but I am happy we started doing this,”
The CDC, which did not respond to a request for comment, long has used a surveillance network to measure flu activity after it occurred. The forecasting effort is expected to help health officials better plan for peak periods of infection, while providing the public, especially those in high-risk groups including the elderly, with more accurate information about when best to avoid traveling, shopping or mingling in large crowds.
CMU’s Delphi-Stat system uses artificial intelligence — machine-learning technology — “to make predictions based on past patterns and on input from the CDC’s domestic influenza surveillance system,” the university said in a release. Its other system, Delphi-Epicast, relies on the “wisdom of the crowds” concept that involved about 100 volunteers who competed week after week with predictions based in part on historic flu data provided to them.
During the past flu season, Delphi-Stat did slightly better on short-term forecasts while Delphi-Epicast did slightly better on long-term forecasts, said Ryan Tibshirani, a Delphi group member and associate professor of statistics and machine learning, in a news statement. Delphi-Epicast finished first with a skill score of 0.451 while Delphi-Stat finished second with 0.438, with “perfect clairvoyance” earning a 1.0. Assumptions about what will happen next, based on an average of what already happened, would produce a score of 0.237.
The Delphi group also is developing a forecasting system for dengue fever, with plans to use its forecasting tools for various diseases including the Human Immunodeficiency Virus, drug resistance, and such epidemic viral infections as Ebola, Zika and Chikungunya.
Mr. Rosenfeld, who holds a Ph.D. in computer science, said it’s too early to make predictions about the upcoming flu season, with the next CDC competition beginning in October.
David Templeton: dtempleton@post-gazette.com or 412-263-1578.
First Published: September 6, 2017, 3:21 p.m.
