Facebook researchers have developed a general AI framework called Recursive Belief-based Learning (ReBeL) that they say achieves better-than-human performance in heads-up, no-limit Texas holdem poker while using less domain knowledge than any prior poker AI. They assert that ReBeL is a step toward developing universal techniques for multi-agent interactions in other words, general algorithms that can be deployed in large-scale, multi-agent settings. Potential applications run the gamut from auctions, negotiations, and cybersecurity to self-driving cars and trucks.
Combining reinforcement learning with search at AI model training and test time has led to a number of advances. Reinforcement learning is where agents learn to achieve goals by maximizing rewards, while search is the process of navigating from a start to a goal state. For example, DeepMinds AlphaZero employed reinforcement learning and search to achieve state-of-the-art performance in the board games chess, shogi, and Go. But the combinatorial approach suffers a performance penalty when applied to imperfect-information games like poker (or even rock-paper-scissors), because it makes a number of assumptions that dont hold in these scenarios. The value of any given action depends on the probability that its chosen, and more generally, on the entire play strategy.
The Facebook researchers propose that ReBeL offers a fix. ReBeL builds on work in which the notion of game state is expanded to include the agents belief about what state they might be in, based on common knowledge and the policies of other agents. ReBeL trains two AI models a value network and a policy network for the states through self-play reinforcement learning. It uses both models for search during self-play. The result is a simple, flexible algorithm the researchers claim is capable of defeating top human players at large-scale, two-player imperfect-information games.
At a high level, ReBeL operates on public belief states rather than world states (i.e., the state of a game). Public belief states (PBSs) generalize the notion of state value to imperfect-information games like poker; a PBS is a common-knowledge probability distribution over a finite sequence of possible actions and states, also called a history. (Probability distributions are specialized functions that give the probabilities of occurrence of different possible outcomes.) In perfect-information games, PBSs can be distilled down to histories, which in two-player zero-sum games effectively distill to world states. A PBS in poker is the array of decisions a player could make and their outcomes given a particular hand, a pot, and chips.
Above: Poker chips.
Image Credit: Flickr: Sean Oliver
ReBeL generates a subgame at the start of each game thats identical to the original game, except its rooted at an initial PBS. The algorithm wins it by running iterations of an equilibrium-finding algorithm and using the trained value network to approximate values on every iteration. Through reinforcement learning, the values are discovered and added as training examples for the value network, and the policies in the subgame are optionally added as examples for the policy network. The process then repeats, with the PBS becoming the new subgame root until accuracy reaches a certain threshold.
In experiments, the researchers benchmarked ReBeL on games of heads-up no-limit Texas holdem poker, Liars Dice, and turn endgame holdem, which is a variant of no-limit holdem in which both players check or call for the first two of four betting rounds. The team used up to 128 PCs with eight graphics cards each to generate simulated game data, and they randomized the bet and stack sizes (from 5,000 to 25,000 chips) during training. ReBeL was trained on the full game and had $20,000 to bet against its opponent in endgame holdem.
The researchers report that against Dong Kim, whos ranked as one of the best heads-up poker players in the world, ReBeL played faster than two seconds per hand across 7,500 hands and never needed more than five seconds for a decision. In aggregate, they said it scored 165 (with a standard deviation of 69) thousandths of a big blind (forced bet) per game against humans it played compared with Facebooks previous poker-playing system, Libratus, which maxed out at 147 thousandths.
For fear of enabling cheating, the Facebook team decided against releasing the ReBeL codebase for poker. Instead, they open-sourced their implementation for Liars Dice, which they say is also easier to understand and can be more easily adjusted. We believe it makes the game more suitable as a domain for research, they wrote in the a preprint paper. While AI algorithms already exist that can achieve superhuman performance in poker, these algorithms generally assume that participants have a certain number of chips or use certain bet sizes. Retraining the algorithms to account for arbitrary chip stacks or unanticipated bet sizes requires more computation than is feasible in real time. However, ReBeL can compute a policy for arbitrary stack sizes and arbitrary bet sizes in seconds.
See more here:
Facebook develops AI algorithm that learns to play poker on the fly - VentureBeat
- Most Poker Rooms Remain Closed As Gambling Markets And Casinos Reopen In The United States - CardPlayer.com [Last Updated On: June 17th, 2020] [Originally Added On: June 17th, 2020]
- New poker rules in the light of the COVID-19 pandemic - Tampa Bay Times [Last Updated On: June 17th, 2020] [Originally Added On: June 17th, 2020]
- Genting Ceases All Live Poker Activity in the United Kingdom - PokerNews.com [Last Updated On: June 17th, 2020] [Originally Added On: June 17th, 2020]
- Ike Haxton Raises Concerns About the Health of Live Poker Players - HighstakesDB [Last Updated On: June 17th, 2020] [Originally Added On: June 17th, 2020]
- King's Casino reopening poker room soon with plenty on the schedule - World Casino Directory [Last Updated On: June 17th, 2020] [Originally Added On: June 17th, 2020]
- Advt: 5 reasons why everyone is hooked to online poker! Have you joined the party yet? - Times of India [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- Orange City OKs more poker players, and Greyhound racing in Daytona is over - The West Volusia Beacon [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- How Coronavirus changed the perception of Poker - CalvinAyre.com [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- Natural8 2020 WSOP Online Hand of the Week: A Wholesome Hand for Ryan Depaulo - PokerNews.com [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- How Has Poker Changed Over Many Years? - Loop21 [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- Trump's approach to politics bears the hallmarks of a bad poker player, author says - CBC.ca [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- Today's lottery news: the Poker Lotto numbers and results for Monday July 20, 2020 are here... - Born2Invest [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- The Deck Is Not Rigged: Poker and the Limits of AI - Undark Magazine [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- Key Points To Consider When Choosing Online Poker Tables - Top10PokerWebsites [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- NJ Special Legislation and the Impact on Online Casino Players in NY - Blog - The Island Now [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- The Phil Ivey Of Poker - HBCU Buzz [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- Venetian poker room going to 8-handed tables - Las Vegas Review-Journal [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- Chasing Poker Greatness: #68 Andrew Brokos: Co-Host ThinkingPoker Podcast & Author of Play Optimal Poker - Pokerfuse [Last Updated On: July 21st, 2020] [Originally Added On: July 21st, 2020]
- FULL-HOUSE VS QUADS in the WSOP Main Event! Could Selbst have gotten away? - PokerNews.com [Last Updated On: August 3rd, 2020] [Originally Added On: August 3rd, 2020]
- Error in partypoker US Online Network PKO Series Finale Results in Nearly $50K Overlay - PokerNews.com [Last Updated On: August 3rd, 2020] [Originally Added On: August 3rd, 2020]
- Why Negreanu thinks Ivey is the best poker player in the world - Comcast SportsNet Chicago [Last Updated On: August 3rd, 2020] [Originally Added On: August 3rd, 2020]
- An Open Letter To The WSOP: Why Mixed Games Are Good For Poker Player Development - Poker News - CardPlayer.com [Last Updated On: August 3rd, 2020] [Originally Added On: August 3rd, 2020]
- What Site Offers the Best Online Poker Bonus in 2020? - PokerNews.com [Last Updated On: August 3rd, 2020] [Originally Added On: August 3rd, 2020]
- Man, 60, in custody in connection to death of professional poker player Susie Zhao - Detroit Free Press [Last Updated On: August 3rd, 2020] [Originally Added On: August 3rd, 2020]
- 60-year-old man arrested in connection with Oakland Co. professional poker player's death - WXYZ [Last Updated On: August 3rd, 2020] [Originally Added On: August 3rd, 2020]
- VIEW: Poker has a lot to offer to the Robinhood generation - CNBCTV18 [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- Key Elements to Consider When Searching for the Best Poker Game Software Development Company - Communal News [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- LOCK, STOCK AND TWO SMOKING BARRELS To A BIG HAND FOR THE LITTLE LADY : Best Poker Movies To Perfect Your Game Plan - The Fan Carpet [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- Casino Games: Online Poker | What Happened To Full Tilt Poker? - GamingTodaySlotsToday [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- Who Are the All-Time Leaders in Poker in North America? - Poker News Daily [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- UPDATE: One Arrested After Poker Game Argument Leads to Machete Attack in Lexington Park | Southern Maryland News Net - Southern Maryland News Net [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- Caribbean Sports - Best Caribbean Poker Tips & Tricks To Win - Caribbean and Latin America Daily News - News Americas [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- Author and former professional poker player Annie Duke on how conspiracy theories gain ground - TechCrunch [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- Poker Market Size and Growth By Leading Vendors, By Types and Application, By End Users and Forecast to 2020-2027 - Scientect [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- When Will Live Poker Rooms Reopen in Pennsylvania? - Play Pennsylvania [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- 5 of the best gambling apps to play real money poker games - European Gaming Industry News [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- Poker - Wikipedia [Last Updated On: August 26th, 2020] [Originally Added On: August 26th, 2020]
- Andre Akkari Brings eSports and Poker Together as Industry Revels in Good News - HighstakesDB [Last Updated On: November 29th, 2020] [Originally Added On: November 29th, 2020]
- Five True-Life Poker Stories Primed To Be Hollywood Blockbusters - PocketFives [Last Updated On: November 29th, 2020] [Originally Added On: November 29th, 2020]
- Christopher Nunnally Wins Pearl River Poker Open $600 Main Event ($57835) - PokerNews.com [Last Updated On: November 29th, 2020] [Originally Added On: November 29th, 2020]
- What Losing 40 Tournaments In A Row Should Tell You About Your Poker Game - Poker News - CardPlayer.com [Last Updated On: November 29th, 2020] [Originally Added On: November 29th, 2020]
- Phil Ivey's Most Memorable Hand that Changed the Course of Poker History - HighstakesDB [Last Updated On: November 29th, 2020] [Originally Added On: November 29th, 2020]
- Legendary poker player goes all in on Bitcoin, liquidates Ether and Monero - Crypto News Flash [Last Updated On: November 29th, 2020] [Originally Added On: November 29th, 2020]
- Poker Club: All Bluff And Bluster | Screen Rant - Screen Rant [Last Updated On: November 29th, 2020] [Originally Added On: November 29th, 2020]
- What Does Another WSOP Main Event Mean for Poker? - PokerNews.com [Last Updated On: November 29th, 2020] [Originally Added On: November 29th, 2020]
- 2020 PokerNews Holiday Gift #4: SimplePoker.com GTO Poker Solvers and Tools - PokerNews.com [Last Updated On: November 29th, 2020] [Originally Added On: November 29th, 2020]
- Why Casino Royale Director Sweated the Poker Scenes Over Any James Bond Action - MovieWeb [Last Updated On: November 29th, 2020] [Originally Added On: November 29th, 2020]
- What Kinds Of Basics Beginners Should Know In Live Poker Gambling? - - VENTS Magazine [Last Updated On: December 26th, 2020] [Originally Added On: December 26th, 2020]
- Poker Legend Maria Ho Details Her Experience on 'The Amazing Race' and 'American Idol' (Exclusive) - PopCulture.com [Last Updated On: December 26th, 2020] [Originally Added On: December 26th, 2020]
- Great minds: 10 players who excel at both chess and poker - chess24 [Last Updated On: December 26th, 2020] [Originally Added On: December 26th, 2020]
- Head of Marketing- Poker Powher - PEAK6 - Built In Chicago [Last Updated On: December 26th, 2020] [Originally Added On: December 26th, 2020]
- 'Twas The Night Before Christmas - Poker Edition - PocketFives [Last Updated On: December 26th, 2020] [Originally Added On: December 26th, 2020]
- Maria Ho Reflects on Hall of Fame Poker Career, Teases Her Future Plans (Exclusive) - PopCulture.com [Last Updated On: December 26th, 2020] [Originally Added On: December 26th, 2020]
- Bitcoin And Crypto Trading Tips From Poker World Champion Annie Duke - Forbes [Last Updated On: December 26th, 2020] [Originally Added On: December 26th, 2020]
- Chemistry Lessons: Building the Perfect High Stakes Poker Cast - PocketFives [Last Updated On: December 26th, 2020] [Originally Added On: December 26th, 2020]
- Poker Stories of the year: Negreanu and Polk's high stakes feud - CalvinAyre.com [Last Updated On: December 26th, 2020] [Originally Added On: December 26th, 2020]
- 2021 Poker Predictions: Will COVID-19 Force a WSOP Cancellation? - CardsChat.com [Last Updated On: December 26th, 2020] [Originally Added On: December 26th, 2020]
- Huck Seed inducted to Poker Hall of Fame - CalvinAyre.com [Last Updated On: January 7th, 2021] [Originally Added On: January 7th, 2021]
- Conor Beresford Wins 2020 Online Poker Player of the Year - PocketFives [Last Updated On: January 7th, 2021] [Originally Added On: January 7th, 2021]
- WPT Readies for Live Poker in Las Vegas and Florida - LegalUSPokerSites.com [Last Updated On: January 7th, 2021] [Originally Added On: January 7th, 2021]
- Run It Once Poker in 2020: Innovation, Promotions and Growth - Pokerfuse [Last Updated On: January 7th, 2021] [Originally Added On: January 7th, 2021]
- Jail for woman who attacked neighbour with poker in Cork town - EchoLive.ie [Last Updated On: January 17th, 2021] [Originally Added On: January 17th, 2021]
- Poker Coming Back to Detroit This Week - USGamblingSites.com [Last Updated On: January 17th, 2021] [Originally Added On: January 17th, 2021]
- This woman used her Psychology Ph.D. to become a poker star - Yahoo News [Last Updated On: January 17th, 2021] [Originally Added On: January 17th, 2021]
- Looking Ahead To The World Series Of Poker Finale - LA Canyon News [Last Updated On: January 17th, 2021] [Originally Added On: January 17th, 2021]
- Sheldon Adelson Died and the Poker Community has Mixed Feelings - CardsChat.com [Last Updated On: January 17th, 2021] [Originally Added On: January 17th, 2021]
- Poker Is on the Cards for Detroit This Friday - GamblingNews.com [Last Updated On: January 17th, 2021] [Originally Added On: January 17th, 2021]
- FIVE THINGS: The Poker Hall of Fame Deserves a Next Level Upgrade - PocketFives [Last Updated On: January 17th, 2021] [Originally Added On: January 17th, 2021]
- Tony G Resigns from Parliament; Will He Head Back to Poker? - PokerNews.com [Last Updated On: January 17th, 2021] [Originally Added On: January 17th, 2021]
- High Stakes Poker S8, E5: You Won't Believe the Pot Tom Dwan Won - PokerNews.com [Last Updated On: January 17th, 2021] [Originally Added On: January 17th, 2021]
- 5 key concepts in poker and chess | chess24.com - chess24 [Last Updated On: January 17th, 2021] [Originally Added On: January 17th, 2021]
- Several States Begin To Lift Casino Closures And Ease Restrictions - CardPlayer.com [Last Updated On: January 29th, 2021] [Originally Added On: January 29th, 2021]
- Jack Hardcastle Wins the WPT Montreal Main Event at partypoker ($447859) - PokerNews.com [Last Updated On: January 29th, 2021] [Originally Added On: January 29th, 2021]
- Los Angeles Poker Rooms Given Green Light to Reopen this Week - CardsChat.com [Last Updated On: January 29th, 2021] [Originally Added On: January 29th, 2021]
- Partypoker US Network Posts Big Online Poker Revenue Gains in New Jersey During the Final Quarter of 2020 - Pokerfuse [Last Updated On: January 29th, 2021] [Originally Added On: January 29th, 2021]
- 888 Executive: US Online Poker to Benefit from Latest Wire Act Ruling - Casino.Org News [Last Updated On: January 29th, 2021] [Originally Added On: January 29th, 2021]
- The FIVES Poker Podcast: WPT in the Headlines, Polk Tilts Negreanu - PocketFives [Last Updated On: January 29th, 2021] [Originally Added On: January 29th, 2021]
- Justice Kagan: Poker Shark - Above the Law [Last Updated On: January 29th, 2021] [Originally Added On: January 29th, 2021]
- Is online poker in the cards for North Dakota? - The Dickinson Press [Last Updated On: January 29th, 2021] [Originally Added On: January 29th, 2021]
- Online poker in the cards for North Dakota? - INFORUM [Last Updated On: January 29th, 2021] [Originally Added On: January 29th, 2021]