Libratus Poker AI beats humans for $1.76mln - how was the computer made?

The Libratus Poker AI, created by Noam Brown et al. at Carnegie Mellon University, has beaten four human professional players at No-Limit Hold'em for a sum of $1,766,250, which amounts to $14.72 taken from human team per single hand.

This is the first first time in history, where an AI machine won with such strong numbers over human professionals in the game of No-Limit Hold'em.

The first time challenge started was two years - back then the program went against four professional players and eventually lost against the humans. Developers of the AI used the past two years to improve the program dramatically and we can see the results.

Dong Kim, Jason Les, Jimmy Chou and Daniel McAulay - those four represented the humans in this challenge. Challenge lasted for 120,000 hands – 30,000 per player - and ran from January 11-30. For each hand the both human player and AI started with 20,000 chips with the blinds at 50/100, which means every single hand was played in a 200 blinds deep scenario.

To reduce the luck and variance factor, two special rules were put in place:
1. All hands were mirrored.
If Player A got aces vs. kings at one table against the AI, Player B received the kings while the AI had aces at other table. This would reduce the variance between each of four games played.
2. No variance over all-ins.
If a hand ended in a all-in situation before the river, no more cards were dealt and each player received his equity in chips. That means 70/30 favourite on turn that went all-in and got called would receive 70% of the pot.

In the end, the computer won $1,766,250, which translated to 14.7 big blinds per 100 hands, which is a very significant result. It's also worth a note, that in the end all four human players lost over their 30,000 hands against Libratus.

Here are the individual results:
Player / Loss / Loss Per Hand
Dong Kim -$85,649 -$2.85
Jimmy Chou -$522,857 -$17.43
Jason Les -$880,087 -$29.34
Daniel McAuley -$277,657 -$9.26

So how Libratus came to exist?
To generate the strategies for all those situations Brown and Sandholm, creators of Libratus, used a supercomputer worth almost $10,000,000 called Bridges. Bridges played against itself day after day, accumulating billions of hands and tried randomly all kinds of different strategies. Basically, generating the strategies was a colossal trial and error run.

In an extensive AMA on Reddit Brown explained the learning process of Libratus like this:
“The basis for the bot is reinforcement learning using a special variant of Counterfactual Regret Minimization. Prior to this competition, it had only played poker against itself. It did not learn its strategy from human hand histories."

How does the mirroring work exactly. Cause explained that way it seems pretty absurd or were the players not able to communicate

On February 02 2017 21:06 Spitfiree wrote: How does the mirroring work exactly. Cause explained that way it seems pretty absurd or were the players not able to communicate

They're not able to communicate. Two players played normally, and their pairs in the mirror were put in the "dungeon" without phones or outside communication. They rotated who was in the dungeon.

thats sick
"Counterfactual Regret Minimization" wtf is this thing

Man is Les really that good of a player? He's gotten dominated in both AI challenges now

could you just freeze at 2000s state, Poker

hmm having got my math degree last year all i understand is about 1/3 of that paragraph.

how this mirroring thing works is this: if dong gets dealt cards against computer, jason gets dealt the cards that the computer got dealt against dong. So in other words if dong runs good, jason will be on the other end of that run good. Stop saying jason is a fish lol he is prob worst on the team but by a very slim margin and certainly one of the best hunl players in the world. Dong ran hot, Jason ran bad. Their combined team results were almost the same as the other team.

On February 03 2017 01:08 Stroggoz wrote: hmm having got my math degree last year all i understand is about 1/3 of that paragraph. how this mirroring thing works is this: if dong gets dealt cards against computer, jason gets dealt the cards that the computer got dealt against dong. So in other words if dong runs good, jason will be on the other end of that run good. Stop saying jason is a fish lol he is prob worst on the team but by a very slim margin and certainly one of the best hunl players in the world. Dong ran hot, Jason ran bad. Their combined team results were almost the same as the other team.

ye exactly :|

The no all-in variance and mirror probably ironically increased variance since lets say you get allin vs libratus KK v AA, you lose 80% stack, but if your mirror buddy slowplay AA on a xxxxK board you lose your whole stack.

On February 03 2017 05:02 Baalim wrote: The no all-in variance and mirror probably ironically increased variance since lets say you get allin vs libratus KK v AA, you lose 80% stack, but if your mirror buddy slowplay AA on a xxxxK board you lose your whole stack.

this is a good point

regardless, i think this is the beginning of the end for online poker boys (at least for non Americans)...it was fun while it lasted

Would a pot limit version of holdem be good or bad for the bot?

What the computer basically does is run out infinite simulations, more than a human can ever do in a 100 lifetimes and was able to take the results in the form of 'regrets'. It was able to quantify the value of each scenario by going through every possible option on every street and seeing which one had the less negative outcome.

Having the advantage of running every possible unique holdem hand scenario 100s of times in your data base will make the computer a favorite everytime. It will have every hand ever played in holdem history in its playbook and will be able to pull up its hands on the fly.

The computer is the one that has really 'seen it all'.

One thing I don't get is the tanking of the bot. A $10'000'000 worth super computer was tanking all the time, while the humans had to play all day long in this slow pace in order to make the necessary hands for the day. No rests, no rest days, playing B game in huge chunks of the time. If this tanking was preprogrammed in order to cut time from humans it's a huge abuse. I have read lots of comments on that and other abusive rules and I am not sure, but perhaps it could be a huge scam performed in order to receive millions of financing.

nlhu is a very computationally complex game, and it doesn't surprise me that it takes a long time to go through the decision tree. The real scam is that military ppl fund this so they can dominate the world better with a super computer coming up with good military strategies.

The future of machines taking human jobs is inevitable. Everything, which boils down to calculation will some day be performed by machines and most interesting will be how far could this go (back to strong ai thread lol). I didn't expected poker to be solved so soon. Gz to the science team!

There were a lot of instances where the bot does different more thin actions for value that the players called off with worse. While the bot did not 'hero' call or was put in as thin of situations

I think one spot was a 3bet or 4bet pot and humans had KQ bot had AK flop like was 1010J Q 9. On the river the Bot CR shove allin and dong or dougie hero called.

The mirror it ended up being ch ch or ch call or bet call

was bound to happpen.

for a group of top pros i'd get better players to play the bot.

would really like hand historys if anyone has them. I also have a runitonce account that lasts for about 15 more days and maybe sauce might make a video analyzing the bot.

On February 03 2017 05:47 TimDawg wrote: Show nested quote + On February 03 2017 05:02 Baalim wrote: The no all-in variance and mirror probably ironically increased variance since lets say you get allin vs libratus KK v AA, you lose 80% stack, but if your mirror buddy slowplay AA on a xxxxK board you lose your whole stack. this is a good point regardless, i think this is the beginning of the end for online poker boys (at least for non Americans)...it was fun while it lasted

it's really only for HU poker though, which is more or less dying everywhere

On February 03 2017 17:59 Fayth wrote: Show nested quote + On February 03 2017 05:47 TimDawg wrote:   On February 03 2017 05:02 Baalim wrote: The no all-in variance and mirror probably ironically increased variance since lets say you get allin vs libratus KK v AA, you lose 80% stack, but if your mirror buddy slowplay AA on a xxxxK board you lose your whole stack. this is a good point regardless, i think this is the beginning of the end for online poker boys (at least for non Americans)...it was fun while it lasted it's really only for HU poker though, which is more or less dying everywhere

You of course would have a better idea than me. I'm just seeing this from a technological advancement standpoint. It seems to me there will be some sort of advanced software with the ability to play more and more GTO as time goes on and I don't see us having much more time before we reach that point.