Translator: Hélène Vernet Reviewer: Bangyou Xiang The rise of the machines! Who here is scared of killer robots? (Laughter) I am! I used to work in UAVs – Unmanned Aerial Vehicles – and all I could think seeing these things is that someday, somebody is going to strap a machine-gun to these things, and they’re going to hunt me down in swarms. I work in robotics at Brown University and I’m scared of robots. Actually, I’m kind of terrified, but, can you blame me? Ever since I was a kid, all I’ve seen are movies that portrayed the ascendance of Artificial Intelligence and our inevitable conflict with it – 2001 Space Odyssey, The Terminator, The Matrix – and the stories they tell are pretty scary: rogue bands of humans running away from super intelligent machines. That scares me. From the hands, it seems like it scares you as well. I know it is scary to Elon Musk. But, you know, we have a little bit of time before the robots rise up.
Robots like the PR2 that I have at my initiative, they can’t even open the door yet. So in my mind, this discussion of super intelligent robots is a little bit of a distraction from something far more insidious that is going on with AI systems across the country. You see, right now, there are people – doctors, judges, accountants – who are getting information from an AI system and treating it as if it is information from a trusted colleague. It’s this trust that bothers me not because of how often AI gets it wrong. AI researchers pride themselves in accuracy on results.
It’s how badly it gets it wrong when it makes a mistake that has me worried. These systems do not fail gracefully. So, let’s take a look at what this looks like. This is a dog that has been misidentified as a wolf by an AI algorithm. The researchers wanted to know: why did this particular husky get misidentified as a wolf? So they rewrote the algorithm to explain to them the parts of the picture it was paying attention to when the AI algorithm made its decision. In this picture, what do you think it paid attention to? What would you pay attention to? Maybe the eyes, maybe the ears, the snout … This is what it paid attention to: mostly the snow and the background of the picture.
You see, there was bias in the data set that was fed to this algorithm. Most of the pictures of wolves were in snow, so the AI algorithm conflated the presence or absence of snow for the presence or absence of a wolf. The scary thing about this is the researchers had no idea this was happening until they rewrote the algorithm to explain itself.
And that’s the thing with AI algorithms, deep learning, machine learning. Even the developers who work on this stuff have no idea what it’s doing. So, that might be a great example for a research, but what does this mean in the real world? The Compas Criminal Sentencing algorithm is used in 13 states to determine criminal recidivism or the risk of committing a crime again after you’re released. ProPublica found that if you’re African-American, Compas was 77% more likely to qualify you as a potentially violent offender than if you’re a Caucasian. This is a real system being used in the real world by real judges to make decisions about real people’s lives. Why would the judges trust it if it seems to exhibit bias? Well, the reason they use Compas is because it is a model for efficiency.
Compas lets them go through caseloads much faster in a backlogged criminal justice system. Why would they question their own software? It’s been requisitioned by the State, approved by their IT Department. Why would they question it? Well, the people sentenced by Compas have questioned it, and their lawsuits should chill us all. The Wisconsin State Supreme Court ruled that compass did not deny a defendant due process provided it was used “properly.” In the same set of rulings, they ruled that the defendant could not inspect the source code of Compass. It has to be used properly but you can’t inspect the source code? This is a disturbing set of rulings when taken together for anyone facing criminal sentencing.
You may not care about this because you’re not facing criminal sentencing, but what if I told you that black box AI algorithms like this are being used to decide whether or not you can get a loan for your house, whether you get a job interview, whether you get Medicaid, and are even driving cars and trucks down the highway. Would you want the public to be able to inspect the algorithm that’s trying to make a decisiom between a shopping cart and a baby carriage in a self-driving truck, in the same way the dog/wolf algorithm was trying to decide between a dog or a wolf? Are you potentially a metaphorical dog who’s been misidentified as a wolf by somebody’s AI algorithm? Considering the complexity of people, it’s possible. Is there anything you can do about it now? Probably not, and that’s what we need to focus on. We need to demand standards of accountability, transparency and recourse in AI systems. ISO, the International Standards Organization, just formed a committee to make decisions about what to do for AI standards. They’re about five years out from coming up with a standard.
These systems are being used now, not just in loans, but they’re being used in vehicles like I was saying. They’re being used in things like Cooperative Adaptive Cruise Control. It’s funny that they call that “cruise control” because the type of controller used in cruise control, a PID controller, was used for 30 years in chemical plants before it ever made into a car. The type of controller that’s used to drive a self-driving car and a machine learning, that’s only been used in research since 2007. These are new technologies. We need to demand the standards and we need to demand regulation so that we don’t get snake oil in the marketplace.
And we also have to have a little bit of skepticism. The experiments in Authority done by Stanley Milgram after World War II, showed that your average person would follow an authority figures orders even if it meant harming their fellow citizen. In this experiment, every day Americans would shock an actor past the point of him complaining about her trouble, past the point of him screaming in pain, past the point of him going silent in simulated death, all because somebody with no credentials, in a lab coat, was saying some variation of the phrase “The experiment must continue.” In AI, we have Milgram’s ultimate authority figure.
We have a dispassionate system that can’t reflect, that can’t make another decision, that there is no recourse to, that will always say “The system or “The process must continue.” Now, I’m going to tell you a little story. It’s about a car trip I took driving across country. I was coming into Salt Lake City and it started raining. As I climbed into the mountains, that rain turned into snow, and pretty soon that snow was whiteout. I couldn’t see the taillights of the car in front of me. I started skidding. I went 360 one way, I went 360 the other way. I went off the highway. Mud-coated my windows, I couldn’t see a thing. I was terrified some car was going to come crashing into me. Now, I’m telling you this story to get you thinking about how something small and seemingly mundane like a little bit precipitation, can easily grow into something very dangerous. We are driving in the rain with AI right now, and that rain will turn to snow, and that snow could become a blizzard.
We need to pause, check the conditions, put in place safety standards, and ask ourselves how far do we want to go, because the economic incentives for AI and automation to replace human labor will be beyond anything we have seen since the Industrial Revolution. Human salary demands can’t compete with the base cost of electricity. AIs and robots will replace fry cooks and fast-food joints and radiologists in hospitals. Someday, the AI will diagnose your cancer, and a robot will perform the surgery. Only a healthy skepticism of these systems is going to help keep people in the loop. And I’m confident, if we can keep people in the loop, if we can build transparent AI systems like the dog/wolf example where the AI explained what it was doing to people, and people were able to spot-check it, we can create new jobs for people partnering with AI.
If we work together with AI, we will probably be able to solve some of our greatest challenges. But to do that, we need to lead and not follow. We need to choose to be less like robots, and we need to build the robots to be more like people, because ultimately, the only thing we need to fear is not killer robots, it’s our own intellectual laziness. The only thing we need to fear is ourselves. Thank you. (Applause) .