Five different AI algorithms have teamed up to kick human butt in Dota 2, a popular strategy computer game.
Recommended for You
Researchers at OpenAI, a nonprofit based in California, developed the algorithmic A team, which they call the OpenAI Five. Each algorithm uses a neural network to learn not only how to play the game, but also how to cooperate with its AI teammates. It has started defeating amateur Dota 2 players in testing, OpenAI says.
This is an important and novel direction for AI, since algorithms typically operate independently. Approaches that help algorithms cooperate with each other could prove important for commercial uses of the technology. AI algorithms could, for instance, team up to outmaneuver opponents in online trading or ad bidding.
Collaborative algorithms might also cooperate with humans. OpenAI previously demonstrated an algorithm capable of competing against top humans at single-player Dota 2. The latest work builds on this using similar algorithms modified to value both individual and team success.
The algorithms do not communicate directly except through game play. “What we’ve seen implies that coordination and collaboration can emerge very naturally out of the incentives,” says Greg Brockman, one of the founders of OpenAI, which aims to develop artificial intelligence openly and in a way that benefits humanity. He adds that the team has tried substituting a human player for one of the algorithms and found this to work very well. “He described himself as feeling very well supported,” Brockman says.
Dota 2 is a complex strategy game in which teams of five players compete to control a structure within a sprawling landscape. Players have different strengths, weaknesses, and roles, and the game involves collecting items and planning attacks, as well as engaging in real-time combat. Pitting AI programs against computer games has become a familiar means of measuring progress.
DeepMind, a subsidiary of Alphabet, famously developed a program capable of learning to play the notoriously complex and subtle board game Go with superhuman skill. A related program then taught itself from scratch to master Go and then chess simply by playing against itself. The strategies required for Dota 2 are more defined than in chess or Go, but the game is still difficult to master.
It is also challenging for a machine because it isn’t always possible to see what your opponents are up to, and because teamwork is required. The OpenAI Five learn by playing against various versions of themselves. Over time, the programs developed strategies much like the ones humans use–figuring out ways to acquiring gold by “farming” it, for instance, as well as adopting a particular strategic role or “lane” within the game.
AI experts say the achievement is significant. “Dota 2 is an extremely complicated game, so even beating strong amateurs is truly impressive,” says Noam Brown, a researcher at Carnegie Mellon University in Pittsburgh. “In particular, dealing with hidden information in a game as large as Dota 2 is a major challenge.” Brown previously worked on an algorithm capable of playing poker, another imperfect-information game, with superhuman skill (see “Why poker is a big deal in AI“). If the OpenAI Five team can consistently beat humans, Brown says, that would be a major achievement in AI.
However, he notes that given enough time, humans might be able to figure out weaknesses in the AI team’s playing style.
Other games could also push AI further, Brown says. “The next major challenge would be games involving communication, like Diplomacy or Settlers of Catan, where balancing between cooperation and competition is vital to success.”
The robot arm is performing a peculiar kind of Sisyphean task. It hovers over a glistening pile of cooked chicken parts, dips down, and retrieves a single piece. A moment later, it swings around and places the chunk of chicken, ever so gently, into a bento box moving along a conveyor belt.
This robot, created by a San Francisco-based company called Osaro, is smarter than any you’ve seen before. The software that controls it has taught it to pick and place chicken in about five seconds–faster than your average food-processing worker. Within the year, Osaro expects its robots to find work in a Japanese food factory.
Anyone worried about a robot uprising need only step inside a modern factory to see how far away that is. Most robots are powerful and precise but can’t do anything unless programmed meticulously. An ordinary robot arm lacks the sense needed to pick up an object if it is moved an inch.
It is completely hopeless at gripping something unfamiliar; it doesn’t know the difference between a marshmallow and a cube of lead. Picking up irregularly shaped pieces of chicken from a haphazard pile is an act of genius. Moreover, until recently, robots have been largely untouched by advances in artificial intelligence.
Over the last five or so years, AI software has become adept at identifying images, winning board games, and responding to a person’s voice with virtually no human intervention. It can even teach itself new abilities, given enough time to practice. All this while AI’s hardware cousins, robots, struggle to open a door or pick up an apple.
That is about to change. The AI software that controls Osaro’s robot lets it identify the objects in front of it, study how they behave when poked, pushed, and grasped, and then decide how to handle them. Like other AI algorithms, it learns from experience.
Using an off-the-shelf camera combined with machine-learning software on a powerful computer nearby, it figures out how to grasp things effectively. With enough trial and error, the arm can learn how to grasp just about anything it might come across.
A robot retrieves products from a bin at Osaro’s headquarters.
Winni Wintermeyer Workplace robots equipped with AI will let automation creep into many more areas of work.
They could replace people anywhere that products need to be sorted, unpacked, or packed. Able to navigate a chaotic factory floor, they might take yet more jobs in manufacturing. It might not be an uprising, but it could be a revolution nonetheless. “We’re seeing a lot of experimentation now, and people are trying a lot of different things,” says Willy Shih, who studies trends in manufacturing at Harvard Business School. “There’s a huge amount of possibility for [automating] repetitive tasks.”
Recommended for You
It’s a revolution not just for the robots, but for AI, too.
Putting AI software in a physical body allows it to use visual recognition, speech, and navigation out in the real world. Artificial intelligence gets smarter as it feeds on more data. So with every grasp and placement, the software behind these robots will become more and more adept at making sense of the world and how it works.
“This could lead to advances that wouldn’t be possible without all that data,” says Pieter Abbeel, a professor at the University of California, Berkeley, and the founder of Embodied Intelligence, a startup applying machine learning and virtual reality to robotics in manufacturing.
Separated at birth
This era has been a long time coming. In 1954, George C. Devol, an inventor, patented a design for a programmable mechanical arm.
In 1961, a manufacturing entrepreneur named Joseph Engelberger turned the design into the Unimate, an unwieldy, awkward machine first used on a General Motors assembly line in New Jersey. From the beginning, there was a tendency to romanticize the intelligence behind these simple machines. Engelberger chose the name “robot” for the Unimate in honor of the androids dreamed up by the science fiction author Isaac Asimov.
But his machines were crude mechanical devices directed to perform a specific task by relatively simple software. Even today’s much more advanced robots remain little more than mechanical dunces that must be programmed for every action. Artificial intelligence followed a different path.
In the 1950s, it set out to use the tools of computing to mimic human-like logic and reason. Some researchers also sought to give these systems a physical presence. As early as 1948 and 1949, William Grey Walter, a neuroscientist in Bristol, UK, developed two small autonomous machines that he dubbed Elsie and Elmer.
These turtle-like devices were equipped with simple, neurologically inspired circuits that let them follow a light source on their own. Walter built them to show how the connections between just a few neurons in the brain might result in relatively complex behavior.
An employee at Embodied Intelligence uses a virtual-reality rig to train a robot.
Courtesy photo But understanding and re–creating intelligence proved to be a byzantine challenge, and AI went into a long period with few breakthroughs.
Meanwhile, programming physical machines to do useful things in the messy real world often proved intractably complex. The fields of robotics and AI began to go their own separate ways: AI retreated into the virtual, while robotics largely measured its progress in terms of novel mechanical designs and clever uses of machines with modest powers of reasoning. Then, about six years ago, researchers figured out how to make an old AI trick incredibly powerful.
The scientists were using neural networks–algorithms that approximate, roughly speaking, the way neurons and synapses in the brain learn from input. These networks were, it turns out, direct descendants of the components that gave Elsie and Elmer their abilities. The researchers discovered that very large, or “deep,” neural networks could do remarkable things when fed huge quantities of labeled data, such as recognizing the object shown in an image with near-human perfection.
The field of AI was turned upside down. Deep learning, as the technique is commonly known, is now widely used for tasks involving perception: face recognition, speech transcription, and training self-driving cars to identify pedestrians and signposts. It has made it possible to imagine a robot that could recognize your face, speak intelligently to you, and navigate safely to the kitchen to get you a soda from the fridge.
One of the first skills that AI will give machines is far greater dexterity. For the past few years, Amazon has been running a “robot picking” challenge in which researchers compete to have a robot pick up a wide array of products as quickly as possible. All of these teams are using machine learning, and their robots are gradually getting more proficient.
Amazon, clearly, has one eye on automating the picking and packing of billions of items within its fulfillment centers.
AI gets a body
In the NoHo neighborhood of New York, one of the world’s foremost experts on artificial intelligence is currently looking for the field’s next big breakthrough. And he thinks that robots might be an important piece of the puzzle. Yann LeCun played a vital role in the deep-learning revolution.
During the 1980s, when other researchers dismissed neural networks as impractical, LeCun persevered. As head of Facebook’s AI research until January, and now as its chief AI scientist, he led the development of deep-learning algorithms that can identify users in just about any photo a person posts. But LeCun wants AI to do more than just see and hear; he wants it to reason and take action.
And he says it needs a physical presence to make this possible. Human intelligence involves interacting with the real world; human babies learn by playing with things. AI embedded in grasping machines can do the same. “A lot of the most interesting AI research now involves robots,” LeCun says.
A remarkable kind of machine evolution might even result, mirroring the process that gave rise to biological intelligence. Vision, dexterity, and intelligence began evolving together at an accelerated rate once hominids started walking upright, using their two free hands to examine and manipulate objects. Their brains grew bigger, enabling more advanced tools, language, and social organization.
Could AI experience something similar?
Until now, it has existed largely inside computers, interacting with crude simulations of the real world, such as video games or still images.
AI programs capable of perceiving the real world, interacting with it, and learning about it might eventually become far better at reasoning and even communicating. “If you solve manipulation in its fullest,” Abbeel says, “you’ll probably have built something that’s pretty close to full, human-level intelligence.”
HBO’s science fiction drama Westworld isn’t just known for its talented cast and its philosophical musings about the nature of reality. It’s also become famous for its reveals, from mind-bending bombshells that link two characters to simple pieces of backstory that bring new insight to a storyline. Watching Westworld is like peeling an onion, one layer at a time.
That’s why for the show’s second season, I’ll be diving into one particular spoilery revelation from each episode, to figure out what it means, how we got here, and where things might go in the episodes to come.
Some weeks, it might be a huge plot twist. In other weeks, it might be something subtle. Either way, we’re going to spoil the hell out of it.
Welcome to the Westworld Spoilers Club.
Heading into the second season finale of Westworld, audiences had already been treated to a subreddit’s worth of twists, reveals, and switcheroos.
Dr. Robert Ford (Anthony Hopkins), it turned out, was still alive(ish), his consciousness uploaded to a vast computer simulation called The Cradle. Delos, Inc. didn’t just buy Westworld because it wanted to get into the theme park business; it wanted to use host technology to create replicants of actual human beings, allowing the company to sell immortality to the highest bidder.
And the Man in Black (Ed Harris), the show’s sturdy antihero, had gone increasingly more mad on his search for “the Door,” until he actually killed his own daughter and began to question whether he was a host himself.
There’s been a lot to unpack, even for a show that embraces puzzlebox sensibilities as much as Westworld. But all that pales in comparison to the amount of narrative sleight of hand and fragmented storytelling that the second-season finale squeezes into its 90-minute running time. “The Passenger” is part philosophy course and part magic trick, with a healthy dose of The Matrix Reloaded thrown in for good measure. It ties up a lot of loose story threads while leaving some major questions unanswered, and does its best to set the stage for the show’s already-announced third season. To top it all off, it ends with the most mind-bending — or is that obnoxious? — post-credits scene in quite some time.
The big reveal?
The Westworld Spoilers Club was built around the idea of digging into one specific reveal from each episode this season, but given how much ground “The Passenger” covers, it’s best to take a big-picture view.
Unwind all the story threads, reshuffle them into some semblance of linear order, and the water-cooler summary goes like this: Bernard makes it to The Valley Beyond, which is home to The Forge, a secret facility where all of the digital copies of the park’s guests have been stored. There he’s met by Dolores (Evan Rachel Wood) and the Man in Black — though the latter is quickly incapacitated. Dolores and Bernard head down to The Forge’s control center, which Bernard discovers also houses a virtual “new world” that the hosts can upload themselves into.
Opening the fabled “Door” that has been discussed all season actually amounts to activating a massive virtual portal in the Westworld park, one that only hosts can see. When they walk through it, the hosts’ minds are uploaded into their new virtual home, where they can be safe from humanity’s meddling.
After the Door is opened, members of the Ghost Nation tribe and other hosts begin uploading themselves in earnest. Dolores would rather burn everything down, however, and in order to stop her from deleting everything and everyone, Bernard shoots and kills her on the spot.
He escapes The Forge, but Dolores already activated the failsafe mechanism that floods the entire Valley Beyond, leading to the aftermath seen in the season premiere.
Back at the Mesa, Elsie (Shannon Woodward) tries to cut a deal with Charlotte Hale (Tessa Thompson), but Hale simply kills Elsie instead. (Unfortunately, Westworld fans, Elsie definitely seems dead this time.) Bernard witnesses the murder, and calls on Ford for help. While he deleted Ford’s code from his own mind in the previous episode, “Vanishing Point,” Ford nevertheless appears, and helps Bernard craft a new host body. It’s later revealed that the host body Bernard builds is none other than a copy of Charlotte Hale herself.
Bernard places Dolores’ consciousness into the Charlotte body, and it kills the real Charlotte and replaces her.
It’s a mindflip. Given the timeline, it forces the audience to realize that half of the time they’ve been watching Charlotte Hale this season, they’ve actually been watching Dolores pretending she’s Charlotte Hale. But the episode is packed with so much business that it doesn’t even bother to revel in the implications of the reveal.
Knowing his memories could give the game away, Bernard decides to scramble his own brain to hide what he’s done.
But first, he has one final conversation with Ford on a beach — during which he realizes that his latest conversations with his mentor have been nothing more than his own imagination. It’s a callback to Dolores recognizing her own internal voice in the season 1 finale, and it serves as evidence that Bernard has finally achieved his own independent consciousness. But he nevertheless moves forward with mucking around with his memories, and he lies down on the sand… which is exactly where Karl Strand (Gustaf Skarsgard) found him in the season premiere, some 14 days after the robot uprising began.
The episode rejoins that 14-days-later timeline as Strand, Hale, and Confused Bernard also head to the Valley Beyond.
They enter The Forge control facility, where they hope to transmit the digital copies of the park’s guests back to the mainland. But Dolores-as-Hale — “Halores,” as one of the show’s in-world websites is calling her — reveals who she really is, kills Strand and the other Delos operatives, and tells Bernard she’s changed her mind about burning everything to the ground. Instead, she uploads Teddy’s consciousness to the virtual world (she grabbed his control unit after he killed himself), and transmits the whole thing to a safe location where Delos, Inc. can never find it.
Then she shoots Bernard in the head.
Using the fact that she looks like a senior Delos board member, Halores then heads out of the park, with five host control units tucked safely in her purse.
She has a brief exchange with Stubbs (Luke Hemsworth), who clearly knows she is a host. He even implies he may be one himself. But he lets her through with the understanding that he will not pursue her in the outside world — and then, finally, Dolores is able to escape Westworld.
What does it mean?
The main story arcs leave the story primed for the third season.
Dolores has escaped, a majority of the hosts are alive in a virtual world somewhere, and even the human characters that died likely exist as digital copies somewhere in The Forge. Maeve, Hector, and Armistice are all gunned down, but there’s a door left open even for them, with Delos lab techs Felix and Sylvester (Leonardo Nam and Ptolemy Slocum, respectively) tasked in the final moments to decide which hosts should be salvaged for the company.
But “The Passenger” isn’t happy with that straightforward ending. After Halores leaves the island, the show cuts back to Bernard, waking up across from Dolores, who’s wearing her season 1 blue dress.
She’s built a new version of him from memory, she says, much as she did the original Bernard, and she believes they will both need to exist in the real world in order for hosts to survive as a species. Her motivations don’t entirely make sense — Dolores goes from wanting to destroy the virtual world to saving it, and from killing Bernard to bringing him back, all without anything substantive happening to change her worldview. She talks about needing him in the way the shlwrunners might talk about needing a protagonist and an antagonist for a story to function, which seems a far cry from her desire to make her own decisions in the real world, away from controlling narratives.
There’s also one small wrinkle.
When Bernard asks Dolores if she escaped, the camera dollies behind his head — and when it comes out the other side, Dolores is suddenly wearing the black cocktail dress she wore the night Arnold talked to her in the real world, back in episode 2, “Reunion.” (Arnold is also suddenly nude, the way a freshly-printed host would be.) This could be written off as a bit of visual metaphor, or of Arnold conflating multiple timelines before finally understanding exactly when and where he is. But during the show’s first season, Westworld pulled this same transitional trick numerous times, usually when it was cutting between different Dolores storylines, sometimes 30 years apart. It doesn’t appear there’s any massive time shift happening in the scene, but it is something worth considering.
Either way, the episode wraps with three hosts in the real world: Dolores, Bernard, and the Charlotte Hale host. Whose mind is currently inside the Charlotte Hale body? That’s left conveniently unaddressed.
Changing the game
If the Dolores / Hale switcheroo wasn’t enough, “The Passenger” also includes a post-credits scene.
Earlier in the episode, The Man in Black took the elevator down into The Forge, but never actually appeared in The Forge. The scene picks that moment up, as he walks out of the elevator… to find the place deserted. It looks like years, even decades, have passed since Dolores and Bernard were in the facility.
The Man in Black is approached by someone that looks exactly like his daughter, Emily (Katja Herbers), and at first, he thinks he’s in another computer simulation.
The woman tells him that’s not the case, that this is what is left of his world. “Tell me, what were you hoping to find?” she asks. “To prove?”
“That no system can tell me who I am,” he says. “That I have a fucking choice.”
“And yet here we are. Again.” The Man in Black quickly realizes the truth: he is a host clone, just another iteration in a long line of them. And she is there to conduct a baseline interview to verify fidelity just as he had done to hundreds of James Delos copies.
The scene is thrilling because it comes out of left field with a major twist, but it’s simultaneously frustrating, because it raises a bunch of questions about the Man in Black’s status throughout the season without providing enough information to actually draw a satisfying conclusion.
Series co-creator Lisa Joy clarified to The Hollywood Reporter that the post-credits scene takes place “in the far, far future,” and that while the incidents portrayed in the season did happen to the human Man in Black, this future host clone has been repeating them on his own loop. It lends credence to the idea that the Man in Black presented in season 2 has actually been this future host version all along. But there are plenty of important questions left unanswered, including what Ford’s intention with the Door game was in the first place.
Given that this was sold as the major hook of the season, it’s a little confounding.
The scene does tie into what is perhaps the most interesting idea in “The Passenger”: that human beings don’t actually have free will at all, and are destined to follow certain patterns based on their makeup. The notion is explained earlier in the episode, when Dolores and Bernard visit the simulated world of The Forge. There, they talk with the computer system that creates the guest copies, who happens to appear as Logan Delos (Ben Barnes).
There’s a lot of exposition — the sequence feels a lot like Neo’s chat with The Architect in The Matrix Reloaded — but the salient point is that The System created millions of virtual copies of James Delos, but every single simulation built to the exact same moment: Delos turning his back on his son when he needed him most.
Just like Delos, no matter what the Man in Black has done in his many host incarnations, he always ends up back at The Forge, having killed his own daughter.
Humans are smart enough to conceive of free will, the show argues, but that’s really just a facade used to interpret the much more basic, fundamental truth. It’s why Dolores is so eager to get to The Valley Beyond in the first place; when she enters The Forge, she looks through the digital copies of all the guests so she can better understand humanity, and defeat it. It’s a “competitive advantage,” as Bernard puts it.
The same idea is also what makes Bernard’s decision to fight back with the Hale host such a big moment for his own evolutionary development.
Over the past few episodes, Ford has pushed Bernard to demonstrate agency and act counter to his more mellow, subservient instincts. Seeing Elsie murdered was enough to push him over the edge, and he took drastic measures he would normally never take — and in the process, obtained his cognitive freedom.
The notion also creates a compelling battlefield for the upcoming third season. Hosts have recognized humans don’t actually have free will, and can be understood like algorithms.
However, the hosts have learned how to change their own primary drives and achieve self-awareness. It could be argued that this is the season where hosts truly evolved past humans.
But none of that matters to this mysterious Man in Black host, running on his loop in the future. In the real world, he couldn’t cope with his failures as a husband and father, and tried to use the park to find a deeper understanding that would bring meaning to his life.
He failed with The Maze. Now, it appears the company’s technology is being used in an attempt for him to posthumously prove that he’s not an unredeemable monster — and he’s failing, loop after loop after loop. That’s a tragic, harrowing idea.
Granted, it’s not actually in the Westworld season finale itself — but in the long wait for season 3, all theories are up for grabs.