RNN & LSTM

What are Recurrent Neural Networks?

Recurrent Neural Networks (RNNs) are a class of neural networks designed for sequential data - data where order matters. Unlike feedforward networks that process each input independently, RNNs maintain a "memory" of previous inputs through hidden states.

Think of reading a sentence: to understand the word "it," you need to remember what came before. RNNs work similarly - they maintain context by passing information from one time step to the next.

Key insight: RNNs have loops that allow information to persist, making them perfect for sequences like text, speech, time series, and video.

Why RNNs Matter

Sequential data is everywhere in real-world applications:

• Natural Language Processing: Text generation, sentiment analysis, machine translation
• Speech Recognition: Converting audio to text (Siri, Alexa)
• Time Series Prediction: Stock prices, weather forecasting, sensor data
• Music Generation: Composing melodies and harmonies
• Video Analysis: Action recognition, video captioning
• Handwriting Recognition: Converting handwritten text to digital

When to Use RNNs/LSTMs

Choose RNNs when:

• Your data has a sequential or temporal nature
• Context from previous inputs is important for predictions
• Input/output lengths can vary (unlike CNNs that need fixed sizes)
• You need to model dependencies over time

Note: For many NLP tasks, Transformers (BERT, GPT) have largely replaced RNNs due to better performance and parallelization. However, RNNs are still valuable for streaming data, online learning, and when computational resources are limited.

How RNNs Work

At each time step, an RNN takes two inputs:

• Current input: x(t) - the data at current time step
• Previous hidden state: h(t-1) - memory from previous steps

It produces:

• Output: y(t) - prediction at current time step
• New hidden state: h(t) - updated memory for next step

The same weights are shared across all time steps, allowing the network to generalize patterns across the sequence.

The Vanishing Gradient Problem

Traditional RNNs struggle with long sequences due to the vanishing gradient problem:

• During backpropagation through time, gradients get smaller and smaller
• The network can't learn long-term dependencies
• Information from many steps ago gets lost

Example: In "The clouds are in the sky," predicting "sky" is easy (short-term). But in a long paragraph, if "clouds" appeared 50 words ago, a vanilla RNN would struggle to remember it.

Solution: LSTM and GRU architectures specifically address this problem.

LSTM: Long Short-Term Memory

LSTM is a special RNN architecture designed to remember information for long periods. It solves the vanishing gradient problem through a sophisticated gating mechanism.

LSTM Components

LSTM has four main components:

• Cell State (C): The "memory highway" that carries information across time steps with minimal changes
• Forget Gate: Decides what information to throw away from cell state (0 = forget, 1 = keep)
• Input Gate: Decides what new information to add to cell state
• Output Gate: Decides what to output based on cell state

LSTM Flow

1. Forget: Look at h(t-1) and x(t), decide what to forget from C(t-1)
2. Input: Decide what new information to store in cell state
3. Update Cell: Update cell state C(t-1) to C(t)
4. Output: Decide what to output from the updated cell state

Building an LSTM with Keras

Example: Text Generation

import numpy as np
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Sample text data
text = """Your training text goes here.
This could be Shakespeare, tweets, or any sequential text."""

# Tokenize text
tokenizer = Tokenizer()
tokenizer.fit_on_texts([text])
sequences = tokenizer.texts_to_sequences([text])[0]

vocab_size = len(tokenizer.word_index) + 1

# Create training sequences
seq_length = 50
X, y = [], []

for i in range(seq_length, len(sequences)):
    X.append(sequences[i-seq_length:i])
    y.append(sequences[i])

X = np.array(X)
y = np.array(y)

# Build LSTM model
model = keras.Sequential([
    layers.Embedding(vocab_size, 100, input_length=seq_length),
    layers.LSTM(150, return_sequences=True),
    layers.Dropout(0.2),
    layers.LSTM(100),
    layers.Dense(100, activation='relu'),
    layers.Dense(vocab_size, activation='softmax')
])

model.compile(
    loss='sparse_categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

# Train
history = model.fit(X, y, epochs=50, batch_size=128, verbose=1)

# Generate text
def generate_text(seed_text, num_words):
    for _ in range(num_words):
        # Tokenize seed text
        encoded = tokenizer.texts_to_sequences([seed_text])[0]
        encoded = pad_sequences([encoded], maxlen=seq_length, truncating='pre')

        # Predict next word
        predicted = model.predict(encoded, verbose=0)
        predicted_id = np.argmax(predicted, axis=-1)[0]

        # Convert to word
        word = tokenizer.index_word.get(predicted_id, '')
        seed_text += ' ' + word

    return seed_text

generated = generate_text("Once upon a time", 50)
print(generated)

LSTM for Time Series Prediction

import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from tensorflow import keras
from tensorflow.keras import layers

# Load time series data
df = pd.read_csv('stock_prices.csv')
data = df['close'].values.reshape(-1, 1)

# Scale data
scaler = MinMaxScaler()
data_scaled = scaler.fit_transform(data)

# Create sequences
def create_sequences(data, seq_length):
    X, y = [], []
    for i in range(len(data) - seq_length):
        X.append(data[i:i+seq_length])
        y.append(data[i+seq_length])
    return np.array(X), np.array(y)

seq_length = 60  # Use 60 days to predict next day
X, y = create_sequences(data_scaled, seq_length)

# Split train/test
split = int(0.8 * len(X))
X_train, X_test = X[:split], X[split:]
y_train, y_test = y[:split], y[split:]

# Build LSTM model
model = keras.Sequential([
    layers.LSTM(50, return_sequences=True, input_shape=(seq_length, 1)),
    layers.Dropout(0.2),
    layers.LSTM(50, return_sequences=False),
    layers.Dropout(0.2),
    layers.Dense(25),
    layers.Dense(1)
])

model.compile(optimizer='adam', loss='mse')

# Train
history = model.fit(
    X_train, y_train,
    epochs=50,
    batch_size=32,
    validation_data=(X_test, y_test),
    verbose=1
)

# Predict
predictions = model.predict(X_test)
predictions = scaler.inverse_transform(predictions)

# Evaluate
from sklearn.metrics import mean_squared_error, mean_absolute_error
mse = mean_squared_error(scaler.inverse_transform(y_test), predictions)
mae = mean_absolute_error(scaler.inverse_transform(y_test), predictions)
print(f"MSE: {mse:.2f}, MAE: {mae:.2f}")

GRU: Gated Recurrent Unit

GRU is a simplified version of LSTM with fewer parameters, making it faster to train while achieving similar performance.

GRU vs LSTM

• Gates: GRU has 2 gates (reset, update) vs LSTM's 3 gates
• Cell State: GRU combines cell state and hidden state
• Speed: GRU trains faster due to simpler architecture
• Performance: Often comparable to LSTM, sometimes better on smaller datasets

Using GRU

from tensorflow.keras import layers

# Replace LSTM layers with GRU
model = keras.Sequential([
    layers.Embedding(vocab_size, 100, input_length=seq_length),
    layers.GRU(150, return_sequences=True),  # Changed from LSTM
    layers.Dropout(0.2),
    layers.GRU(100),  # Changed from LSTM
    layers.Dense(100, activation='relu'),
    layers.Dense(vocab_size, activation='softmax')
])

model.compile(
    loss='sparse_categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

Bidirectional RNNs

Bidirectional RNNs process sequences in both directions (forward and backward), useful when future context helps understand current input.

Example: In "I like ___," you might predict "pizza" from past context. But in "I ___ pizza," if you see "pizza" ahead, you know the blank is likely a verb like "ate" or "ordered."

from tensorflow.keras.layers import Bidirectional

model = keras.Sequential([
    layers.Embedding(vocab_size, 128, input_length=max_length),
    Bidirectional(layers.LSTM(64, return_sequences=True)),
    Bidirectional(layers.LSTM(32)),
    layers.Dense(64, activation='relu'),
    layers.Dense(num_classes, activation='softmax')
])

Sentiment Analysis with LSTM

from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Load IMDB dataset
vocab_size = 10000
max_length = 200

(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=vocab_size)

# Pad sequences
X_train = pad_sequences(X_train, maxlen=max_length)
X_test = pad_sequences(X_test, maxlen=max_length)

# Build model
model = keras.Sequential([
    layers.Embedding(vocab_size, 128, input_length=max_length),
    layers.LSTM(128, dropout=0.2, recurrent_dropout=0.2),
    layers.Dense(1, activation='sigmoid')
])

model.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['accuracy']
)

# Train
history = model.fit(
    X_train, y_train,
    epochs=5,
    batch_size=64,
    validation_data=(X_test, y_test),
    verbose=1
)

# Evaluate
loss, accuracy = model.evaluate(X_test, y_test)
print(f"Test Accuracy: {accuracy:.4f}")

Best Practices

• Sequence Length: Experiment with different sequence lengths; longer isn't always better
• Normalization: Scale numerical sequences (especially time series) to [0,1] or standardize
• Dropout: Use dropout (0.2-0.5) to prevent overfitting; LSTM has special recurrent_dropout
• Batch Size: Smaller batches (32-128) often work better for sequences
• Return Sequences: Set return_sequences=True when stacking RNN layers
• Gradient Clipping: Use gradient clipping to prevent exploding gradients
• Start Simple: Try GRU before LSTM; it's often sufficient and trains faster
• Consider Transformers: For NLP tasks with sufficient data, Transformers often outperform RNNs

RNN vs CNN vs Transformer

Common Pitfalls

• Data Leakage: Don't shuffle time series data; maintain temporal order
• Overfitting: RNNs overfit easily on small datasets; use dropout and regularization
• Exploding Gradients: Use gradient clipping (clip_norm or clip_value)
• Stateless vs Stateful: Most use cases need stateless RNNs; stateful is for continuous streaming
• Wrong Padding: Use 'pre' padding for right-aligned sequences, 'post' for left-aligned

Related Articles