Skip to content
ShilpBot

AI Fundamentals · Lesson 2 of 2

How Neural Networks Learn: Layers, Weights and Backpropagation

A visual, jargon-light explanation of how neural networks work: neurons, weights, loss, gradient descent and backpropagation, plus the problem of overfitting.

By ShilpBot Team · 8 min read · Updated

A neural network is a mathematical function made of many small units called neurons, loosely inspired by the brain. By adjusting millions of numbers inside it, the network learns to map inputs, like image pixels, to outputs, like the label "cat".

The building blocks

Neurons and weights

Each artificial neuron takes several numbers as input, multiplies each by a weight that says how important that input is, adds them up, adds a bias, and passes the result through an activation function. The activation function introduces non-linearity, which is what allows a network to learn curved, complex patterns rather than only straight-line relationships.

Layers

Neurons are arranged in layers:

  • The input layer receives the raw data, such as pixel values.
  • Hidden layers transform the data step by step. Networks with many hidden layers are called deep networks.
  • The output layer gives the final answer, such as probabilities for each class.

Learning in four steps

Training repeats the same cycle many times.

  1. Forward pass: feed an example through the network and get a prediction.
  2. Loss calculation: a loss function measures how wrong the prediction was. A bigger number means a worse guess.
  3. Backpropagation: the network works backwards to calculate how much each weight contributed to the error.
  4. Weight update: gradient descent nudges every weight slightly in the direction that reduces the loss.

Gradient descent in one picture

Imagine standing on a foggy hillside and trying to reach the valley floor. You cannot see the whole landscape, but you can feel which way the ground slopes under your feet. You take a small step downhill, feel again, and repeat. The size of each step is the learning rate. Steps that are too large overshoot the valley, and steps that are too small take forever.

Key training terms

TermMeaning
EpochOne full pass through the training dataset
BatchA small group of examples processed together
Learning rateHow big each weight update is
LossA number measuring how wrong the model is
ParameterA learned number, such as a weight or bias

Overfitting and how to reduce it

A network with enough capacity can memorise its training examples, including their noise, and then fail on new data. Signs of overfitting are low training error with much higher test error. Common remedies include:

  • Getting more and more varied training data
  • Using a simpler model
  • Dropout, which randomly switches off neurons during training
  • Early stopping, which halts training when test performance stops improving

Different networks for different data

  • Convolutional neural networks (CNNs) specialise in images by scanning for local patterns like edges and shapes.
  • Recurrent networks and transformers handle sequences such as text, audio and sensor streams.

Robots commonly use CNN-based models to interpret camera images, which we explore next in Computer Vision for Robots.

Key takeaways

  • A network is layers of weighted sums plus activation functions.
  • Learning means adjusting weights to reduce a loss, using backpropagation and gradient descent.
  • Always judge a model on data it has not seen during training.

Frequently asked questions

What does backpropagation do?

Backpropagation calculates, efficiently and layer by layer from the output back to the input, how much each weight contributed to the prediction error. Gradient descent then uses that information to update the weights.

How many layers make a network deep?

There is no strict cutoff. Networks with more than a couple of hidden layers are usually called deep, and modern models can have dozens or hundreds of layers.

Why do neural networks need so much data?

They contain a very large number of adjustable weights. Without enough varied examples they tend to memorise the training set instead of learning general patterns.