Origin of Deep Learning
Recognizing a beautiful butterfly sitting on a flower in your garden, different shapes of clouds in the sky, a cup of tea on your table, the words on this paper, or many different kind of things that we see in our daily lives seem to be an easy task for us.
A human brain does this within a fraction of second even if they are appearing in different poses and in different environments. In this context the obvious question that arises is that how the brain does recognizes such varied patterns or how does it classify these objects based on their appearance? Finding the answers to some of these questions led to the evolution of the interesting concept of Artificial Neural Networks (ANN’s).
ANN’s derives its inspiration from the biological brain. ANN’s provide one of the most powerful classes of solutions for machine learning.
1943: a logician, Walter Pitt and, a neuroscientist, Warren McCulloch describes a piece of puzzle in 1943, when they made the first neural network based on mathematical model. They generate a combination of mathematics and algorithms that aim to capture human brain thought processes.
1950: Alan turing invents the ‘Turing Test' to get that a computer can process real thoughts. To pass this test, a computer should able to make dumb a human and give believe that a computer work like a human.
1952: Arthur Samuel implemented learning program for computer. It was a game of checkers, and this game was improved by IBM with playing and studying it more and makes some strategies to win the game.
1957: A psychologist ‘Rosenblatt’ presented a paper titled “The Perceptron: A Perceiving and Recognizing Automaton” at Laboratory of Cornell Aeronautical. He said that he would “create an electromechanical system that would learn to recognize identities between different patterns of electrical information, in a manner that closely related to functionality of a biological brain.”
1959: Neurophysiologists and Nobel Laureates David H. Hubel finds out two types of cells in the main visual cortex, simple cells and complex cells. Various artificial neural networks are initialized with these cells in one or another way, which is not a goal for deep learning; it is definitely a heavily influenced field.
1960: Aerospace and ocean engineer ‘Kelley’ at the Polytechnic Institute of Virginia presented a paper titled “Gradient Theory of Optimal Flight Paths”, it was a widely recognized paper in his field. He presented various ideas about control theory such as, the system behavior with inputs, and how to modify that behavior with feedback; these ideas were directly implemented to AI and ANNs over the years.
1965: A Mathematician ‘Ivakhnenko’ and his associates Lapa arguably invented first deep learning network in 1965, and applies theories and ideas on that point. ‘Ivakhnenko’ invented Group Method of Data Handling (GMDH) which described as a “family of inductive algorithms for computer which based on mathematical modeling of multi-parameter datasets that provides fully automatic and parametric optimization of models” and applies it to neural networks. His learning algorithms used deep feed forward multilayer perceptron’s using statistical methods at each layer to find the best features and forward them through the system. Using GMDH, Ivakhnenko was able to create an 8-layer deep network in 1971, and he successfully demonstrated the learning process in a computer identification system called Alpha.
1967: Invented “nearest neighbor” algorithm in 1967, that allows basic pattern recognition for computers. This algorithm used to find out path for traveling salesmen problem, where a salesman starts journey from a random city and visits all cities with covering shortest path.
1979: A best known and recognized neural network innovator ‘Fukushima’ created an artificial neural network known as, Neocognitron that knows how to recognize or find visual patterns. It was used for recognizing hand written characters, recommendation systems, and for processing natural language. ‘Fukushima’ work was heavily affected Hubel and Wiesel that led to the invention of the first ‘convolutional neural network’, which based on the visual cortex found in animals. This network is a variation of multilayer perceptron that created to use minimal amounts of preprocessing.
1982: In 1982, Hopfield created Hopfield Networks. Hopfield Networks are the recurrent neural networks that serves as a content addressable memory system, and they led to the invention of a popular implementation tool in the 21st century for deep learning.
1985: In 1985, A Computational neuroscientist ‘Terry Sejnowski’ used learning process to create ‘NETtalk’. This program mainly pronounces English words in same way as a child does and this program was improved over time with converting text to speech.
1989: Watkins in 1989 published a paper titled “Learning from Delayed Rewards”. In this paper, he described the Q-learning concept, which led to improved reinforcement learning practically and feasibility.
1993: Schmidhuber, A computer scientist from Germany in 1993, solve out a deep learning problem for which he used 1,000 layers in the recurrent neural network. It was a change in the neural network study.
1997: In 1997, Schmidhuber and Hoch Reiter proposed a long short-term memory (LSTM) recurrent neural network framework. This network improved recurrent neural network efficiency by eliminating long-term dependency.
2006: In 2006, ‘Geoffrey Hinton’ introduced the term ‘deep learning’ to study new algorithms that improves computers to ‘see’ and differentiate texts and objects.
2009: Fei-Fei Li, a professor at artificial intelligence lab in Stanford University, introduced Image Net . It is a large, 14 million labeled images database provide access to researchers, educators, and students.
2011: Alex Krizhevsky in 2011 developed AlexNet, and with this he won several machine and deep learning competitions. AlexNet is an improved version of LeNet5. It has eight layers, five convolutional layers with three fully connected layers. It uses rectified linear units to improve the dropout and speed.
2014: Researcher ‘Ian Goodfellow’ and his team developed Generative Adversarial Networks (GAN) in 2014. This model has two networks, the first takes data and creates in- differentiable samples, while the second one receives the created samples and data and determines that each data point is genuine or not.
2017: Now Many companies are offering powerful, machine and deep learning products. They have done deep learning tasks in a fraction of the time by using neural network developed by Microsoft on XC50 supercomputers with graphic unit 1000 NVidia Tesla P100.
Introduction
Deep learning is the subpart of machine learning, utilizing neural networks that mimic the structure and functionality of the human brain. Deep learning is a modern approach to data analysis and prediction, and has recently become very popular.
In certain applications, deep learning has experienced much greater success than machine learning, one of the reasons for this is that machine learning is not able to efficiently process vast volumes of data as well as not being able to automatically extract the features from the data.
Deep learning is recognized as an important method for artificial intelligence science with diverse applications in many fields such as speech recognition, object recognition and image classification.
Deep learning is broken down into five types:
- Deep Neural Network
- Convolutional Neural Network
- Artificial Neural Network
- Recurrent Neural Network
- Q-learning
These forms of deep learning are growing rapidly with numerous packages of applications, including Theano, CuDNN, Caffee and Keras.
In particular, in the classification of images, deep learning is favored because it can provide efficient results. For image classification, the computer is equipped with large image datasets in deep learning and then it adjusts the pixel value of the image to an internal representation where patterns on the input image can be identified by the classifiers. The Convolutional Neural Network (CNN) is the most widely used form of deep learning for the classification of images. CNN adopts data directly from the image, thus removing the manual extraction function.
Why Deep Learning ?
While operating with large quantities of data machine learning is not sufficient, as it cannot solve artificial intelligence problems such as processing natural languages and recognizing an object. Deep learning is capable of solving different problems, since it requires a little programmer help to concentrate primarily on the individual features by itself. Both Machine learning and deep learning are used to train a computer and to properly classify the data.
Machine learning algorithms manually find the appropriate features such as, edges or corners to train machine learning model for data classification. Below figure (a) shows how a machine learning model works.
 |
| figure (a) Machine Learning Model [http://cs231n.stanford.edu/] |
 |
| figure (b) Deep Learning Model [http://cs231n.stanford.edu/] |