Peering inside an AI’s brain will help us trust its decisions – New Scientist

Is it a horse?

Weegee(Arthur Fellig)/International Center of Photography/Getty

By Matt Reynolds

Oi, AI what do you think youre looking at? Understanding why machine learning algorithms can be tricked into seeing things that arent there is becoming more important with the advent of things like driverless cars. Now we can glimpse inside the mind of a machine thanks to a test that reveals which parts of an image an AI is looking at.

Artificial intelligences dont make decisions in the same way that humans do. Even the best image recognition algorithms can betricked into seeing a robin or cheetahin images that are just white noise, for example.

Its a big problem, says Chris Grimm atBrown Universityin Providence, Rhode Island. If we dont understand why these systems make silly mistakes, we should think twice abouttrusting them with our livesin things like driverless cars, he says.

So Grimm and his colleagues created a systemthat analyses an AI to show which part of an image it is focusing onwhen it decides what the image is depicting. Similarly, for a document-sorting algorithm, the system highlights which words the algorithm used to decide which category a particular document should belong to.

Its really useful to be able to look at an AI and find out how its learning, says Dumitru Erhan, a researcher at Google. Grimms tool provides a handy way for a human to double-check that an algorithm is coming up with the right answer for the right reasons, he says.

To create his attention-mapping tool, Grimm wrapped a second AI around the one he wanted to test. This wrapper AI replaced part of an image with white noise to see if that made a difference to the original softwares decision.

If replacing part of an image changed the decision, then that area of the image was likely to be an important area for decision-making. The same applied to words. If changing a word in a document makes an AI classify a document differently, it suggests that word was key to the AIs decision.

Grimm tested his technique on an AI trained to sort images into one of 10 categories, including planes, birds, deer and horses. His system mapped where the AI was looking when it made its categorisation. The results suggested that the AI had taught itself to break down objects into different elements and then search for each of those elements in an image to confirm its decision.

For example, when looking at images of horses, Grimms analysis showed that the AI first paid close attention to the legs and then searched the image for where it thought a head might be anticipating that the horse may be facing in different directions. The AI took a similar approach with images containing deer, but in those cases it specifically searched for antlers. The AI almost completely ignored parts of an image that it decided didnt contain information that would help with categorisation.

Grimm and his colleagues also analysed an AItrained to play the video game Pong. They found that it ignored almost all of the screen and instead paid close attention to the two narrow columns along which the paddles moved. The AI paid so little attention to some areas that moving the paddle away from its expected location fooled it into thinking it was looking at the ball and not the paddle.

Grimm thinks that his tool could help people work out how AIs make their decisions. For example, it could be used to look atalgorithms that detect cancer cells in lung scans,making sure that they dont accidentally come up with the right answers by looking at the wrong bit of the image. You could see if its not paying attention to the right things, he says.

But first Grimm wants to use his tool to help AIs learn. By telling when an AI is not paying attention, it would let AI trainers direct their software towards relevant bits of information.

Reference: arXiv, arxiv.org/abs/1706.00536