Category: AI Foundations

Introduction: The Power of LSTMs

The Long Short-Term Memory (LSTM) network is a specialised kind of Recurrent Neural Network (RNN) architecture, designed specifically to solve the problem of vanishing gradients that plagues traditional RNNs when dealing with long sequences of data.

While standard RNNs struggle to retain information from many steps ago, LSTMs are engineered with a dedicated Cell State (C_t)—often called the “conveyor belt”—that runs straight through the network. This Cell State is regulated by three distinct, multiplicative gates (Forget, Input, and Output) that learn to selectively remember or forget information, allowing the network to capture and utilise long-term dependencies in sequential data like text, speech, and time series. The mathematical equations below illustrate how these gates precisely control the flow of both long-term memory (C_t) and short-term output (h_t).

---

LSTM Cell Architecture Equations

This breakdown translates the LSTM diagram into its corresponding mathematical equations, showing exactly how the inputs (x_t, h_t-1, C_t-1) are processed to generate the outputs (h_t, C_t).

The σ symbol represents the Sigmoid function, and W and b represent the weight matrices and bias vectors learned during training.

1. The Gates (Control)

The first step is calculating the three gates, each using the current input (x_t) and the previous hidden state (h_t-1) and applying a sigmoid function (σ):

A. Forget Gate (f_t):

f_t = σ(W_f · [h_t-1, x_t] + b_f)

Purpose: Decides which information to forget from the old cell state (C_t-1).

B. Input Gate (i_t):

i_t = σ(W_i · [h_t-1, x_t] + b_i)

Purpose: Decides which values to update in the cell state.

C. Candidate Cell State (ᶜ_t):

ᶜ_t = tanh(W_C · [h_t-1, x_t] + b_C)

Purpose: Creates a vector of potential new values that could be added to the cell state.

2. Cell State Update (The Memory)

The Cell State (C_t) is the core memory of the LSTM, updated by combining the old memory and the new candidate memory:

C_t = f_t ∗ C_t-1 + i_t ∗ ᶜ_t

• The term ft * Ct-1 implements the forgetting mechanism: the old memory C_t-1 is scaled down by the Forget Gate f_t.
• The term it * ᶜt implements the input mechanism: the new candidate information ᶜ_t is scaled by the Input Gate i_t.
• These two parts are then added to create the new long-term memory, C_t.

3. Hidden State Output (The Prediction)

The Hidden State (h_t) is the final output of the cell at this time step. It is based on the new Cell State, filtered by the Output Gate:

A. Output Gate (o_t):

o_t = σ(W_o · [h_t-1, x_t] + b_o)

Purpose: Decides which parts of the (squashed) Cell State will be exposed as the Hidden State.

B. Final Hidden State (h_t):

h_t = o_t ∗ tanh(C_t)

• The new Cell State C_t is passed through tanh to bound the values between -1 and 1.
• The result is then element-wise multiplied by the Output Gate o_t to produce the final short-term memory and output vector, h_t.

---

Conclusion: The Importance of Selective Memory

The LSTM architecture, as described by these equations, fundamentally improved the capability of recurrent neural networks to model complex dependencies over long sequences. By using three learned, multiplicative gates to regulate the flow into and out of the Cell State, the LSTM is able to maintain a stable, uncorrupted memory path, overcoming the practical limitations of standard RNNs.

This innovation has made LSTMs essential tools in areas requiring deep contextual understanding, leading to breakthroughs in speech recognition, machine translation, and text generation, before the wider adoption of the Transformer architecture.

Next Steps

Interested in a simpler alternative? Check out the GRU (Gated Recurrent Unit), which combines the forget and input gates into a single update gate—achieving similar performance with fewer parameters. For cutting-edge sequence modeling, explore how Transformers use attention mechanisms to process entire sequences in parallel, bypassing recurrence altogether.

Recurrent Neural Networks: Processing Sequences

RNNs were designed for sequential data – text, time series, audio, and video. Unlike feedforward networks that process fixed-size inputs, RNNs maintain a hidden state that acts as memory, allowing information to persist across the sequence and enabling context-aware processing.

At each timestep, the hidden state combines the previous state with new input through learned transformations. This recurrence creates a computational graph that unfolds through time, theoretically allowing information from early inputs to influence later processing indefinitely.

In practice, vanilla RNNs struggle with long sequences due to vanishing gradients. When backpropagating through many timesteps, gradients shrink exponentially, causing the network to forget early information. Exploding gradients present the opposite problem, causing training instability.

While largely superseded by Transformers for most applications, understanding RNNs remains valuable. They introduced key concepts like sequence modeling and temporal dependencies. LSTM and GRU variants solved the gradient problems, and some real-time applications still benefit from RNNs streaming nature.

Convolutional Neural Networks: How AI Sees

CNNs revolutionized computer vision by mimicking how the visual cortex processes images. Small learnable filters slide across the image detecting features, with early layers finding edges and later layers identifying complex objects. This hierarchical feature learning made accurate image recognition possible.

Key components include convolutional layers where filters detect local patterns, pooling layers that reduce spatial dimensions while preserving important features, and fully connected layers that combine features for final classification. The architecture dramatically reduces parameters through weight sharing.

CNNs achieve translation invariance – they detect features regardless of position in the image. A cat in the corner is recognised the same as one in the centre. This property emerges naturally from the sliding filter approach and makes CNNs robust to object placement.

Famous architectures include LeNet (1998), AlexNet (2012) which sparked the deep learning revolution, VGG demonstrating depth matters, ResNet enabling 100+ layer networks with skip connections, and modern EfficientNets balancing accuracy and efficiency. Each advanced our understanding of what makes CNNs effective.

Activation Functions: The Key to Non-Linearity

Without activation functions, neural networks would be limited to linear transformations no matter how many layers they have. Activations introduce non-linearity, enabling networks to learn complex patterns like image recognition and language understanding that linear models cannot capture.

ReLU (Rectified Linear Unit) outputs max(0,x) – simple, fast, and surprisingly effective. It has become the default for hidden layers. Sigmoid squashes output to 0-1, useful for binary classification but prone to vanishing gradients. Tanh outputs -1 to 1, zero-centred which sometimes helps training.

For output layers, the choice depends on your task. Sigmoid for binary classification, softmax for multi-class (outputs sum to 1 as probabilities), and linear for regression. Modern variants like GELU and Swish offer slight improvements in specific contexts.

Understanding activations helps diagnose training issues. Dead ReLU neurons that never activate, saturated sigmoids causing vanishing gradients, and numerical instability all relate to activation choice. Experimentation within established guidelines usually yields good results.

Backpropagation: The Learning Algorithm

Backpropagation is the algorithm that makes deep learning possible. It efficiently computes how much each weight in a neural network contributed to the prediction error, enabling targeted updates that improve performance. Without backprop, training deep networks would be computationally infeasible.

The algorithm applies the calculus chain rule to propagate error gradients backward through the network. Starting from the output layer, it calculates local gradients then multiplies by downstream gradients to determine each weights contribution to the loss. This recursive computation handles arbitrarily deep networks.

The beauty of backpropagation lies in its efficiency. It computes all gradients in a single backward pass through the network, achieving O(n) complexity. Computing each gradient independently would require O(n squared) forward passes, making training prohibitively slow for modern architectures.

Understanding backprop illuminates common training issues. Vanishing gradients occur when gradients shrink exponentially through layers. Exploding gradients cause instability. Techniques like gradient clipping, proper initialisation, and batch normalisation address these issues while preserving backprops fundamental efficiency.

Deep Learning: Layers of Abstraction

Deep learning derives its name from having many layers, but depth accomplishes more than just size. Each layer builds increasingly abstract representations, transforming raw inputs into meaningful features. This hierarchical learning mirrors how the visual cortex processes information from simple edges to complex objects.

In image recognition, early layers detect edges and simple patterns. Middle layers combine these into textures and shapes. Deeper layers recognise parts like eyes or wheels. Final layers identify complete objects and scenes. This progression from concrete to abstract happens automatically through training.

Deeper networks can represent more complex functions with fewer parameters than shallow networks. However, training deep networks historically faced the vanishing gradient problem where gradients became infinitesimally small in early layers. Innovations like ReLU activation and residual connections solved this, enabling networks with hundreds of layers.

Understanding layer-wise abstraction helps in architecture design and debugging. Visualising intermediate activations reveals what each layer learns. Transfer learning exploits this by reusing early general-purpose layers while fine-tuning later task-specific ones.

Transformers: The Architecture Behind Modern AI

The Transformer architecture, introduced in the landmark 2017 paper Attention Is All You Need, revolutionized artificial intelligence. It powers GPT, BERT, and virtually every modern language model. Unlike previous sequential models, Transformers process entire sequences simultaneously, enabling unprecedented parallelisation and long-range dependency modeling.

The key innovation is self-attention, allowing each element to attend to all others regardless of distance. Components include the encoder for understanding input and decoder for generating output. Multi-head attention runs multiple attention mechanisms in parallel, each learning different relationship types like syntax, semantics, and coreference.

Transformers win because they are highly parallelisable across GPUs, can model dependencies across thousands of tokens, and scale predictably with more data and parameters. This scaling behaviour led to the emergence of capabilities in large language models that surprised even their creators.

Understanding Transformer architecture is essential for modern AI development. Whether fine-tuning pre-trained models or building custom solutions, knowing how attention mechanisms, positional encodings, and feed-forward layers interact enables effective model design and debugging.

How Machine Learning Actually Learns

Machine learning is optimisation, not magic. The learning process iteratively adjusts model parameters to minimise a loss function measuring prediction errors. This systematic approach revolutionized intelligent systems, enabling computers to improve through experience rather than explicit programming.

The training loop follows a consistent pattern. During the forward pass, input flows through the model producing predictions. The loss function calculates deviation from targets. During backpropagation, gradients indicate how each parameter contributed to error. Finally, the optimiser updates weights to reduce loss. This cycle repeats millions of times.

Gradient descent drives optimisation. Imagine finding the lowest point in a landscape while blindfolded – you feel the slope and step downhill. Gradient descent follows the steepest path to minimum error. Learning rate controls step size: too large overshoots, too small gets stuck.

Modern optimisers like Adam and RMSprop improve upon basic gradient descent by adapting learning rates and using momentum to escape local minima. Understanding these fundamentals enables diagnosing training issues, tuning hyperparameters, and designing efficient architectures for any scale.

The Digital Brain: Biological vs Artificial

Artificial Intelligence attempts to replicate human cognition in machines. Both biological and artificial neurons share fundamental principles: receiving multiple inputs, having activation thresholds, and strengthening connections through repeated use. However, implementation differs dramatically between organic brain computing and silicon-based AI processing.

The human brain contains 86 billion neurons connected by trillions of synapses, operating massively parallel while consuming only 20 watts. Training large AI models requires megawatts. Biological neurons communicate through electrochemical signals with complex temporal dynamics. Artificial neurons use simplified mathematical functions on numerical inputs.

Current AI lacks consciousness and genuine understanding. These systems recognise patterns and generate statistically likely outputs, but do not truly comprehend meaning. Large language models write poetry and solve problems without subjective experience or awareness. This distinction defines both AI capabilities and limitations.

Artificial General Intelligence matching human cognitive abilities across domains remains distant and uncertain. While narrow AI excels at specific tasks, replicating human flexibility and creativity presents challenges we barely understand. The brain-AI comparison inspires while humbling us about how much remains to learn.

What is a Neural Network?

A neural network is a computational model inspired by the biological structure of the human brain. At its core, it consists of interconnected nodes called neurons, organised into distinct layers that process information hierarchically. These artificial neurons receive inputs, apply mathematical transformations, and produce outputs that feed into subsequent layers, creating a powerful system capable of learning complex patterns.

The architecture consists of three fundamental layer types. The input layer receives raw data such as images or text. Hidden layers transform inputs through weighted sums and activation functions, gradually learning meaningful representations. The output layer produces the final result – a classification, prediction, or generated content.

Each connection between neurons carries a weight determining influence. During training, weights are adjusted using backpropagation to minimise prediction errors. The computation follows: output equals activation function applied to sum of inputs times weights plus bias. This simple operation repeated millions of times enables learning incredibly complex patterns.

Understanding this architecture is crucial for AI development. The depth and width of networks determine learning capacity. Proper initialisation, regularization, and optimisation ensure successful training whether building image classifiers, language models, or recommendation systems.