PyTorch Tensors and Autograd: Shapes, Gradients and Backpropagation

Tensor Essentials

PyTorch tensors are multidimensional arrays that can run on CPU or GPU. They store model inputs, outputs, labels, weights, gradients, and intermediate activations. If you understand tensors, shapes, dtypes, devices, and broadcasting, most PyTorch code becomes much easier to debug.

Autograd is PyTorch automatic differentiation. When tensors have `requires_grad=True`, PyTorch records operations on them and builds a dynamic computation graph. Calling `backward()` computes gradients that optimizers use to update model parameters.

A tensor has shape, dtype, device, and values. Shape tells you the dimensions, dtype tells you numeric type, and device tells you where the tensor lives. Shape mismatches are among the most common beginner errors in PyTorch.

Use `.shape` to inspect dimensions before passing tensors into a model.
Use floating tensors for neural network inputs and parameters.
Use long integer tensors for class labels passed to `CrossEntropyLoss`.
Move both model and tensors to the same device with `.to(device)`.
Use `view`, `reshape`, `permute`, and `unsqueeze` deliberately when changing shapes.

Create and Inspect Tensors

import torch

x = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
labels = torch.tensor([0, 1], dtype=torch.long)

print(x.shape)      # torch.Size([2, 2])
print(x.dtype)      # torch.float32
print(labels.dtype) # torch.int64
print(x.device)     # cpu

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
x = x.to(device)
print(x.device)

Autograd Flow

Autograd tracks operations only when at least one participating tensor requires gradients. Model parameters normally require gradients automatically. Data tensors usually do not need gradients unless you are doing special optimization on inputs.

Call `loss.backward()` to compute gradients.
Read gradients from `parameter.grad` after backward.
Call `optimizer.zero_grad()` before the next backward pass.
Use `torch.no_grad()` or `torch.inference_mode()` during evaluation and inference.

Manual Gradient Example

import torch

w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(1.0, requires_grad=True)

x = torch.tensor(3.0)
y_true = torch.tensor(10.0)

y_pred = w * x + b
loss = (y_pred - y_true) ** 2

loss.backward()

print("loss:", loss.item())
print("dLoss/dw:", w.grad.item())
print("dLoss/db:", b.grad.item())

Broadcasting and Shape Bugs

Broadcasting lets PyTorch combine tensors with compatible shapes, but accidental broadcasting can hide bugs. Always verify prediction and target shapes before computing loss.

For regression, predictions and targets should usually have the same shape.
For classification with `CrossEntropyLoss`, logits are shaped `[batch, classes]` and labels are shaped `[batch]`.
Use assertions in training code while developing.

Training Step with Correct Gradient Handling

import torch
from torch import nn

model = nn.Linear(4, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

features = torch.randn(8, 4)
targets = torch.randn(8, 1)

predictions = model(features)
loss = loss_fn(predictions, targets)

optimizer.zero_grad()
loss.backward()
optimizer.step()

print("loss:", loss.item())

Before you move on