CONVOLUTIONAL LAYER
Convolution
Let the input to our neural network has shown in Figure 3.1. It could be an image (color image of CIFAR-10 with a width & height of 32 x 32 pixels and a depth of 3) or a video (a video with a width & height resolution and a frame depth) as shown in [“Understanding of a Convolutional Neural Network ”, Antalya ,Turkey, ICE 2017]. Let's say now, the network receives raw pixels as data. To this end connect the input layer to a single neuron (for example, in the multilayer Perceptron's hidden layer), the CIFAR-10 dataset must have connections of weight of 32x32x3.
 |
| Figure 3.1 Features learn from a Convolutional Neural Network [Saad ALBAWI , Tareq Abed MOHAMMED , Saad AL-ZAWI , “Understanding of a Convolutional Neural Network ”, Antalya ,Turkey, ICE 2017] |
 |
| Figure 3.2 3-D Input in CNN [https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/] |
Now for add more neuron to the hidden layer, we will need another 32x32x3 weight ratio which will become 32x32 parameters in total. For binding the input to only two nodes, over 6000 weight parameters are used to make it simpler Two neurons could not fit any useful processing for an application for image classification. To make it more precise, we can relate the input image to the height and width of the neurons in the next layer with exactly the same values .It can be assumed that this network, such as the edge in the image, is added to the method of processing. however, the aforementioned network requires connections of 32x32x3 by 32x32 by weight, which is (3,145,728).
 |
| Figure 3.3 Convolution as feature for fully connected network [Saad ALBAWI , Tareq Abed MOHAMMED , Saad AL-ZAWI , “Understanding of a Convolutional Neural Network ”, Antalya ,Turkey, ICE 2017] |
Therefore, searching for a more efficient way, instead of a whole partnership, it has emerged that it is a good idea to check for local regions in the image instead of the whole image. Figure 3.3 shows a regional link to the next layer. In other words, only inputs from the respective portion of the previous layer are obtained by the secret neurons in the next layer; for instance, they can only bind to 5x5 neurons. Therefore, if we had 32x32 neurons in the next layer, we would have 5x5x3 by 32x32 links, which is 76,800 (for optimum connectivity, compared to 3,145,728).
While the link size was significantly reduced, there are still too many parameters to resolve. Another simplification concept is to hold the local relationship weights defined for the next layer's complete neurons. This will bind the neighboring neurons with exactly the same weight in the next layer to the local area of the previous layer. Then, reduce several additional parameters and reduce the weight to just 5x5x3 = 75 for connecting neurons from 32x32x3 to 32x32 in the next layer.
Those basic assumptions have many benefits. First, the number of connects in the given example is decreasing from around 3 million to just 75. Other and a more interesting concept is that setting the local connection weights is equivalent to sliding a hierarchical 5x5x3 window into input neurons and mapping the output generated to the correct location. Provides the ability to recognize and detect features no matter where they are in the picture. Therefore they are called convolutions.
Next Figure shows what would happen if we manually select the link weight in a 3x3 window to demonstrate the amazing effect of the convolution matrix.
It can set the matrix to detect the edges of the image, as we can see in Figure.3.4. These matrices are often called a filter, since in image processing they behave like the classic filters. However, these filters are initialized; the training protocol is accompanied by shape filters, which are more appropriate for the given task.
 |
| Figure 3.4 Effects of various matrix convolutions |
After the input layer, it is possible to add more layers to make this method more useful. Several filters can be combined to the sheet. Hence we can extract different features from the image given. Figure 3.5 illustrates how the various layers bind together. -- The layer has its own filter and therefore extracts from the data different characteristics. The neurons show in Figure 3.5 use another filter, but look at the same part of the image data.
 |
| Figure 3.5 Multiple layers which correspond to different filters, but which looking at the same region in the image [https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/] Stride CNN has more chances to decrease the parameters more and more while reducing some of the side effects. One such alternative is the phase of. In the example above, it is clearly assumed that when looking at the regions, the next layer node has several neighbors’ overlaps. We can change the overlap by adjusting the development; Figure. 3.6 shows a 7x7 image. Every time we filter a node, we can only have a 5x5 output. Notice the performance of the three left-in matrices Figure |
3.6, they overlap (and 3 from the middle and three from the right together). If we transfer and make each stride 2 though, then the performance will be 3x3. [3] As shown in Figure 3.7, equation ( 1) formalizes the output size O, given the NxN dimension of the image and the filter size of the FxF.
𝟎 = 𝟏 + ( ( 𝑵−𝑭 ) / 𝑺 ) ---------------------------------------(1)
N = size of the input,
F = size of the filter ,
S = size of the stride.
 |
Figure 3.6 stride-1, Filter window moves one time for every connection [https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/] |
The lack of knowledge that might occur at the edge of the picture is one of the drawbacks of the convolution process. And they never get a chance to be seen except when the filter slides are shot. The use of zero fill is a very easy but efficient approach to solving the problem. Another benefit of zero-padding is managing the production size. As in Figure 3.6, the output will be 5x5 with N = 7 and F = 3 and step 1 (which will be reduced from a 7x7 input). However the output will be 7x7 when applying zero padding, which is exactly the same as the original input (the real N is now 9, use formula (1). The amended formula which includes zero padding is the formula (2).
𝟎=𝟏+ ( (𝑵+𝟐𝑷−𝑭) / 𝑺 ) _____________________(2)
P = number of the layers of the zero-padding
(if P=1 in Figure. 3.8), This concept of padding helps us prevent the size of network performance from shrinking with depth. Therefore, any number of deep convolutional networks can be had.
 |
| Figure 3.8 Zero-padding |
Feature of CNNs
Translations of invariance add to the model the exchange of weights. No matter what the spatial properties are, it helps to filter the learning feature. If this improves performance, they can learn to detect the edge by beginning random values for the filters (as in Figure. 3.4). It's important to remember that if we need to know that something is spatially important in the given input, then using a Shared weight is an extremely bad idea. In addition, this definition can be extended to many dimensions. If it's sequential data like audio, for instance, then you can use one-dimensional audio. If it is an image as shown, it can be applied to two-dimensional convolutions, and for videos or 3D images, a three-dimensional convolution can be used. In the 2012 ImageNet competition, this basic principle outperformed all classical computer vision object recognition approaches, as seen in Figure 3.9.
 |
| Figure 3.9 Visualization of Convolutional deep neural network layers ["DeepFace: Closing the Gap to Human-Level Performance in Face Verification", Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato and Lior Wolf ranzatog@fb.com] |
Convolutional Formula
Formula (3) measures the convolution for one pixel for a further next layer.
𝒏𝒆𝒕(𝒊,𝒋)=(𝒙∗𝒘).[𝒊,𝒋]=Σ .Σ𝒙[𝒎,𝒏]𝒘[𝒊−𝒎,𝒋−𝒏]𝒏𝒎 __________________________(3)
Where the performance of the next layer is net (i, j),' x' is the input image and 'w' is the kernel or filter matrix and '*' is the mechanism of convolution. Figure 3.10 illustrates how convolution works. The input and core element by product of the element are added as can be seen, and the corresponding point is reflected in the next layer.
 |
| Figure 3.10 Details on Convolution layer |
ReLU LAYER ( NONLINEARITY )
After convolution, the next stratum is nonlinearity. To change or decrease the output generated, non-linearity can be used. This layer is used to saturate the output or to limit the generated output.
Sigmoid and tanh have been the most common nonlinearity for many years. Figure. 3.11,which demonstrates common forms of non-linearity. However, the Rectified Linear Unit (ReLU) was used more commonly recently for the following reasons.
1. ReLU has simpler functional as well as gradient descriptions.
ReLU (x) = max (0,x) _____________________________ (4)
𝒅𝒅𝒙 𝑹𝒆𝑳𝑼(𝒙)={𝟏 𝒊𝒇 𝒙>0;0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒} _____________________(5)
 |
| Figure 3.11 Types of Non-Linearity |
2. Saturated function such as sigmoid and tanh creates problems with backward propagation. If the neural network architecture is broader, the gradient signal appears to vanish, which is called the "gradient of disappearance" This happens because, except in the middle, the gradient of such functions is near zero almost everywhere. ReLU does have a steady gradient for the positive input, however. As the function is not distinguishable, it may be ignored by the actual implementation.
3. The ReLU provides a sparser depiction as the zero in the gradient results in a complete zero. Sigmoid and tanh however still have non-zero gradient effects, which may not be ideal for setting up.
POOLING LAYER
The core concept of grouping is down sampling to reduce the difficulty of additional layers. The closest thing to lowering the resolution can be considered in the domain of image processing. Filter numbers do not impact grouping. One of the most common grouping strategies is full grouping. The picture is split into sub-region rectangles and returns only the maximum value of that sub-region 's interior .One of the max. pool sizes most widely used is 2x2. When the 2x2 blocks in the upper left (pink area) are grouped together, as you can see in Figure 3.12, it moves 2 and focuses on the upper right segment. This implies that in classification, Stage 2 is included. You may use Stride 1 to stop sampling down which is not natural. Sampling down should be perceived as not preserving the information's position. This should therefore only be applied where the presence of information (instead of spatial information) is relevant. Additionally, pooling can be used to increase performance with unequal filters and steps. A maximum 3x3 group with step 2, for example, maintains some overlaps between the areas.
 |
| Figure 3.12 It indicates max-pooling. The 2x2 filters and 2 stride max-pooling leads to down-sampling of each 2x2 block being mapped to 1 block (pixel) |
FULLY-CONNECTED LAYER
The completely connected layer is similar to how neurons are organized by a traditional neural network. Consequently, in both the previous and the next layers, each node in a fully connected layer is directly connected to each node, as seen in Figure 3.13 from this figure we can see that each node in the last frames in the grouping layer is linked as a vector to the first layer of the fully connected layer. These are the parameters most widely used for CNN inside these layers, which take a long time to train.
The key disadvantage of a fully connected layer is that it requires multiple parameters in the training examples that require complex calculations. So we are trying to minimize the number of links and nodes. Using the abandon technique will satisfy the nodes and the link removed. LeNet and AlexNet, for example, built a large and deep network by keeping the computational complex constant.
 |
| Figure 3.13 Fully-Connected layer |
CNN Working
A computer understands an image using numbers at each pixels. In Figure 3.14 considered that a black pixel will have value 1 and a white pixel will have -1 value. (Inspired from [http://cs231n.stanford.edu/] & [https://desktop.arcgis.com/en/arcmap/latest/extensions/spatial-analyst/image-classification/what-is-image-classification-.html] )
Figure 3.14 Grey Image of “X” & “O” in CNN [28]
 |
| Figure 3.15 Features/Filters in Convolutional Layer [6] |
Now as Figure 3.16 move the features/filters to every possible position on the images.
1. Line up feature and the image.
2. Multiply each image pixel by the corresponding feature pixel.
3. Add them up
4. Divide by total number of pixel in the feature
Now to keep track of where that feature was, create a map and put the value of the filter at that place.
 |
| Figure 3.16 Creating a Map to Put the Values of the Filter in Convolutional Layer |
Similarly, it will move the feature to every other position of the images and will see how the feature matches that area. Similarly, it will perform the same convolution with every other filters .Finally get an output as shown in Figure 3.17.
 |
| Figure 3.17 Convolution Layer Output |
 |
| Figure 3.18 Output for All Features after using ReLU Layer |
 |
| Figure 3.19 Output for All Features after using Pooling Layer |
 |
| Figure 3.20 Stacking up the Layers (Convolution, ReLU, Pooling) |
 |
| Figure 3.21 Stacking up the Layers and use Fully Connected layer |
When we feed in, ‘X’ and ‘O’. Then there will be some element in the vector that will be high. Consider the image below , as you can see for ‘X’ there are different elements that are high and similarly, for ‘O’ we have different that are high.
Because of the bigger value the input image is classified as ‘X’.
 |
| Figure 3.22 Activation function used in CNN (SOFTMAX FUNCTION) [Shin-Jye Lee, Tonglin Chen, Lun Yu, Chin-Hui Lai , “Image Classification Based on the Boost Convolutional Neural Network on” , 2169-3536 (c) 2018 IEEE] |
 | ||
| Figure 3.23 Working of CNN for Image Classification (training CNN using CIFAR-10 | Datasets) |
In above figures shows how CNN works and Identifies images. There are many datasets available i.e. CIFAR-10, MNIST, MS-COCO, ImageNet, VisualQA, Sentiment140 etc. Here we consider CIFAR-10 dataset for classifying Dog’s image.