• News
  • Learning in Machines & Brains

Computers learn by playing with blocks

by Dan Falk Apr 7 / 16

When an infant plays with wooden blocks, it’s not just playing – it’s also learning about the physical world by interacting with it, pushing and pulling and poking at its various parts. Now a team of artificial intelligence researchers at Facebook are trying to get a computer program to understand its environment in the same way.

The program learns by looking at images of stacked blocks – some of which will topple.

The program, called a convolutional neural network, learns by looking at hundreds of thousands of computer-generated images of stacked blocks – some of them stable, some unstable. If the stack is unstable, the network sees where the blocks end up after the tower topples over. When it’s then exposed to new images – either computer-generated images or pictures of real-life wooden blocks – the network does a remarkably good job of predicting whether the stacks are stable, and if not where the various blocks will end up.

On the computer-generated data, the network performs better than humans do, says Rob Fergus (New York University, Facebook AI Research), a senior fellow in CIFAR’s Learning in Machines & Brains program. “And on real-world data, it’s almost as good as humans,” he says.

Importantly, the network is never taught the rules of mechanics, or of gravity – it simply watches and learns, similar to the way a child learns.

“I don’t think we have Newton’s laws baked into our genes,” says Fergus. “It’s much more likely that we learn by trying things out. When you’re an infant, you play with these blocks, and you get an instinctive feeling. ‘The tower’s about to tip over. The top block is going to fall over, to the left.’ That kind of thing.”

For an infant, playing with blocks may seem like a game, but the child is actually learning how to take in a visual scene and predict what’s going to happen next. And that, in turn, depends on “a kind of physical common sense,” Fergus says. “At the moment, machines don’t have that.”

Networks like the kind Fergus works on still need to be told what to pay attention to. But some day a machine capable of completely unsupervised learning may be possible, he says, although that is probably still far off.

“That is still an unsolved problem,” Fergus says.