CMU's computer poker win over humans was 'statistically significant'
January 31, 2017 3:02 PM
Jason Les, of Costa Mesa , California, competes in the "Brains vs. Artificial Intelligence" at the "Rivers" Casino on the North Shore.
By Sean D. Hamill / Pittsburgh Post-Gazette
The human players already knew it, but the Carnegie Mellon University team that built the computer algorithm, nicknamed Libratus, that beat them at poker over the last three weeks confirmed this morning that the victory was not just sweet, it was “statistically significant.”
At an awards ceremony at Rivers Casino, where the four human players who took part in the 20-day Brains vs. AI poker competition received their performance-based pay, the CMU team said it needed to win $7.70 per hand played for the victory to be deemed statistically significant and accepted as a result in the scientific community.
“But we won $14.70 per hand” over the 120,000 hands played, said Noam Brown, the CMU Ph.D. student who did much of the work on the algorithm with CMU professor Tuomas Sandholm. “That’s pretty good.”
Even more impressive, the first time the CMU team played the world’s best poker players at Heads-Up, No-Limit, Texas Hold ’em, the four human players then beat the prior algorithm, nicknamed Claudico, by a margin that was almost statistically significant for the humans.
To flip around in 20 months “is a huge improvement in a relatively short time,” Mr. Brown said.
That data will take on added significance this coming weekend when Mr. Brown and Mr. Sandholm present a scientific paper on one of the most important parts of Libratus — the “endgame solver” algorithm — at the Association for the Advancement of Artificial Intelligence annual conference in San Francisco.
The conference, where most of the world’s leading AI researchers will gather, will turn into a celebration for the CMU team, said Eric Jackson, a private researcher whose computer-playing poker bot, known as Slumbot, finished second to CMU’s team in a bot-versus-bot competition last year.
“In general the AI community has noticed this result,” said Mr. Jackson in a phone interview from San Francisco. “It’s a surprisingly large jump from being pretty significantly beaten the last time.”
Not only the CMU team, but Rivers Casino was overjoyed by the three-week match, which gave the casino coverage around the world among poker and AI fans.
Craig Clark, Rivers’ general manager, said data on Twitch — the site that live-streamed the competition — showed that people from more than 150 countries either watched or live chatted at least some of the event, compared with about 80 countries in 2015.
“The rematch generated a lot of excitement,” he said.
For the players, it also meant a pretty nice payday — even if they they could have earned even more in a human-to-human poker tournament.
“Not bad for three weeks work,” said player Jimmy Chou, who won $44,438 in final calculation from the $200,000 pot the humans were playing for based on their performance, even though each of them “lost” to Libratus.
Mr. Chou finished third in the competition, ahead of Jason Les (who took home $20,000), but behind Daniel McAulay ($61,212) and the best human player, Dong Kim ($74,388).
“I think the worst result for the human team here would have been walking away thinking, ‘We could have done more,’ ” said Mr. Les. “But as Dr. Sandholm said, ‘You guys tried everything you could.’ ”
The computer was just better, Mr. Les said.
The poker hands were played using theoretical dollars, not real money. A bright-red digitized scoreboard kept daily track of how far ahead Libratus was each day. The final count said that Libratus beat the four humans by $1,766,250.
During his remarks at today’s press conference, Mr. Les said he gave one final look at the scoreboard when he came in, and he took note, with dismay, of the $1.7 million they lost to Libratus.
Then, turning to Mr. Sandholm with a smile, he said: “This is probably going to be the only bad news you are going to get today, Dr. Sandholm: We’re not paying.”
Sean D. Hamill: email@example.com or 412-263-2579 or Twitter: @SeanDHamill
To report inappropriate comments, abuse and/or repeat offenders, please send an email to
firstname.lastname@example.org and include a link to the article and a copy of the comment. Your report will be reviewed in a timely manner.