>The networks are not inefficient, they classify an image in 2ms and state of the art nets run real-time on your iPhone. Companies don't use "enormous numbers of servers" (I assume in training time?) to accomplish these tasks, they use a few dozen GPUs.
He's talking about the training time. Google Brain used 16,000 CPUs and had a training set of 10 million images back in 2012[1]. It is no doubt substantially bigger now.
Actually 2012 was when GPU training was just taking off. A team from the University of Toronto entered one of the larger competitions and won by a large margin by using GPU training. They used 2 GTX 580 over the course of 6 days to train their network on millions of images.
He's talking about the training time. Google Brain used 16,000 CPUs and had a training set of 10 million images back in 2012[1]. It is no doubt substantially bigger now.
[1]http://www.nytimes.com/2012/06/26/technology/in-a-big-networ...