Understanding Neural Networks and Their Components

In the age of deep learning and generative AI, neural network models are at the forefront of innovation. We use neural networks to build deep learning applications for tasks like image recognition, text classification, and speech recognition. Understanding the basics of neural networks is important for anyone interested in artificial intelligence, as it provides the foundation for building complex deep learning models.

Let’s discuss what a neural network is and what its components are. We will break down the structure and key components of a neural network to help you gain a clear understanding of how they work.

What is a neural network?

Imagine you are trying to teach a child to identify numerical digits. Initially, the child might make mistakes identifying the correct numbers. However, if you continue showing the images of the digits to the child and pointing out the correct digits, the child will get better at recognizing them.

A neural network is like a simplified version of the human brain that also learns by example. When we train it using a dataset, it learns the underlying patterns in the dataset and can use them to make predictions or decisions in the future.

Structure of a neural network

Each neural network consists of multiple layers of artificial neurons that process the data to generate the outputs. At a high level, every neural network can be depicted using the following diagram:

In the diagram, each circle represents an artificial neuron that processes the input data using a combination of summation and an activation function. The entire neural network structure is divided into three layers.

Layers in a neural network

A neural network consists of 3 layers, i.e., input layer, hidden layers, and output layer.

Input layer

The input layer is the first layer of a neural network model. As the name suggests, this layer takes in the input data. It has the same number of neurons as the values in the input data vectors.

• If the input data contains 1-D vectors of length N, the input layer has N neurons.
• If the input data contains 2-D arrays of shape MXN, as in images, the input layer will have MxN neurons.

Hidden layers

Hidden layers are the neural network layers between the input and output layers. Each hidden layer can have any number of neurons, often determined by experimentation during model training. The number of hidden layers is also variable and depends on the complexity of the problem we are trying to solve.

Output layer

The output layer is the last layer of a neural network model. It generates the final results, such as probabilities for classification tasks or a numeric value for regression tasks. For classification tasks, the output layer has the same number of neurons as the number of target labels. For regression tasks, the output layer contains a single neuron.

Working of a neural network model

Neural networks are trained by supervised learning, i.e., we give multiple example inputs to the neural network and the associated outputs while training. During model training, the neural networks learn the underlying patterns in the input data using feedforward and backpropagation mechanisms.

Feedforward

The neurons at the input layer process the input data and forward it to the hidden layers. The neurons in each hidden layer process the data and forward their output to the next layer. Finally, the neurons in the output layer receive the processed data from the hidden layers and produce the output. This entire process is called the feedforward mechanism.

Backpropagation

Once the output layer generates a value, the neural network compares the generated output with the actual output in the dataset and calculates the error. This error is passed to the hidden layers from the output layer. The neurons in each layer calculate the partial derivative of the error with respect to their internal parameters like weights and biases. This helps the neurons identify how much the weights and biases affect the error. Accordingly, the neurons update their weights and biases to reduce errors using an optimization algorithm. This entire process is called backpropagation.

Multiple iterations of feedforward and backpropagation mechanisms modify the parameters of neurons in a way that helps the neural network model learn the patterns in the input data.

So, neural networks consist of several layers of neurons, and the neurons update their parameters, like weights and biases, to learn the patterns in the input data. However, what does an artificial neuron look like?
Let’s discuss the structure of neurons, which are the building blocks of artificial neural networks.

Understanding neurons in a neural network

A neuron is the basic building block of an artificial neural network. It mimics a biological neuron. The structure of an artificial neuron is represented by the following diagram:

In this diagram,

• x₁, x₂, and x₃ represent the inputs of the neuron. There is only one input for the neurons in the input layer. In the output and hidden layers, each neuron can receive inputs from any number of neurons in the previous layer.
• The weights w₁, w₂, and w₃ are associated to inputs x₁, x₂, and x₃ respectively. The weight of an input in a neuron determines the importance of the input in deciding the output.
• The value b is the bias associated with the neuron. The bias allows a neuron to change its output independently of the input values, helping the neuron improve flexibility in learning the patterns in the data.

The weights and biases are the internal parameters of the neurons that are adjusted while training the neural network model to learn the patterns in the data. The neuron first calculates the weighted sum of the inputs along with the bias to derive the value z = x₁w₁ + x₂w₂ + x₃w₃ + b. Here, the value z has a linear relation with the inputs. We can vary z by modifying the weights and bias.

However, the linear combination doesn’t capture complex patterns in the input data very well. Even if we combine multiple layers of neurons with linear outputs, the final output of the neural network model will still be a linear combination of the input values, restricting the model from learning complex non-linear patterns in the training data. Therefore, each neuron has an activation function that makes the neuron output non-linear with respect to the inputs.

• The activation function g transforms the weighted sum z into a non-linear output y using functions such as sigmoid and softmax, enabling the neuron to learn and represent complex non-linear patterns.
• The value y is the final value produced by the neuron. It is passed to the next layer as an input if the neuron is present in a hidden layer or input layer. If the neuron is in the output layer, y represents the model output.

Understanding neural network components

While training a neural network model and producing outputs, the neurons use activation functions, loss functions, and optimizers to generate outputs, calculate errors, and adjust the weights and biases for improvement. Let’s discuss these components of neural networks.

Activation function

As discussed in the previous section, activation functions make the neuron output non-linear with respect to the inputs, which enables the neural network to learn complex patterns in the input data. Depending on the problem we are trying to solve, we can use different activation functions, such as the sigmoid function, hyperbolic tangent (tanh), softmax, and rectified linear unit (ReLU).

In general, we can use the following rules to select activation functions in neural networks:

• The rectified linear unit (ReLU) activation function is mainly used in hidden layers.
• We use the sigmoid function to define the output layer for binary classification models.
• For multiclass classification models, we use the softmax function in the output layer.
• For regression models, we use the linear activation function.

Loss function

While training a neural network, we use the loss function to calculate the difference between the actual output and the predicted value of the neural network model.

As a general guideline, we can use the following rules to select a loss function for training a neural network model:

• We use binary cross-entropy to define the loss function for binary classification tasks.
• Categorical cross-entropy is best suited for training multiclass classification models.
• Mean absolute error (MAE) and mean squared error (MSE) are used for regression tasks.

You can read more about the above loss functions in the official keras documentation.

Optimizers

The neural network uses optimizers to update the weights and biases of the neurons during the backpropagation mechanism. There are different types of optimizers, such as Stochastic Gradient Descent (SGD), Adam (Adaptive Moment Estimation), RMSProp, etc. You can get a list of all the optimizers defined in TensorFlow in the documentation.

Metrics

Metrics aren’t used in any aspect of model training. We use metrics to evaluate the model performance once the neural network is trained. For classification tasks, we use metrics like accuracy, binary cross-entropy, categorical cross-entropy, f-1 score, etc., to evaluate the model performance. We can use mean squared error (MSE), mean absolute error (MAE), root mean squared error (RMSE), etc., for regression tasks.

Conclusion

Understanding neural networks and their components is the foundation of building advanced deep learning models. By exploring their core structure and the role of each element in the training process, we’ve examined how neural networks make decisions and solve complex problems.

To deepen your understanding, you can build a neural network model using TensorFlow or you can build a neural network model using PyTorch.

Happy Learning!