PyTorch Sequence Models and Transformers

Sequence Data Shape

Sequence models process ordered data such as text, time series, audio frames, events, and tokens. PyTorch supports recurrent models such as RNNs, GRUs, and LSTMs, and modern attention-based transformer models.

Transformers are widely used because self-attention lets a model relate each token to other tokens in the sequence. Instead of processing tokens strictly one at a time, a transformer can learn contextual relationships across the sequence more efficiently.

Most sequence models work with three dimensions: batch size, sequence length, and feature size. For text, feature size is often an embedding dimension. For time series, it may be the number of measurements at each time step.

Batch size is how many examples are processed together.
Sequence length is how many tokens or time steps each example has.
Embedding dimension is the vector size used to represent each token.
Padding and masks are needed when sequences have different lengths.

Embedding Token IDs

import torch
from torch import nn

token_ids = torch.tensor([
    [1, 5, 9, 0],
    [1, 7, 3, 4],
])

embedding = nn.Embedding(num_embeddings=10, embedding_dim=8, padding_idx=0)
vectors = embedding(token_ids)

print(vectors.shape)  # [batch=2, sequence=4, embedding=8]

Transformer Encoder

A transformer encoder reads a sequence and produces contextual vectors. These vectors can be pooled for classification, used for token tagging, or passed to another model component.

Self-attention learns relationships between positions in the sequence.
Positional information is needed because attention alone does not know token order.
Masks prevent the model from attending to padding tokens or future tokens in causal tasks.
Layer normalization and feed-forward blocks stabilize representation learning.

Small Transformer Encoder Classifier

import torch
from torch import nn

class TextClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, num_classes):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=embed_dim,
            nhead=4,
            batch_first=True,
        )
        self.encoder = nn.TransformerEncoder(encoder_layer, num_layers=2)
        self.classifier = nn.Linear(embed_dim, num_classes)

    def forward(self, token_ids):
        padding_mask = token_ids == 0
        x = self.embedding(token_ids)
        x = self.encoder(x, src_key_padding_mask=padding_mask)
        pooled = x[:, 0]  # simple CLS-style first-token pooling
        return self.classifier(pooled)

model = TextClassifier(vocab_size=5000, embed_dim=64, num_classes=3)
batch = torch.randint(1, 5000, (8, 20))
logits = model(batch)
print(logits.shape)  # [8, 3]

Training Concerns

Sequence models can overfit or become expensive quickly. Start small, validate shapes, use masks correctly, and compare against a simple baseline before increasing layers, heads, and embedding size.

Use `CrossEntropyLoss` for multi-class sequence classification.
Track validation accuracy, F1, or task-specific metrics.
Use gradient clipping for recurrent models and some unstable training runs.
Watch memory usage as sequence length increases.

Classification Loss

labels = torch.tensor([0, 2, 1, 1, 0, 2, 0, 1])
loss_fn = nn.CrossEntropyLoss()

loss = loss_fn(logits, labels)
loss.backward()

print(loss.item())

Before you move on