ai-magazineSearch terms, articles and models

AI Education

PyTorch: What It Is, Its Real History, and How to Use It

By · · 7 min read

Infographic: PyTorch's lineage from the 2002 Torch library through Torch7 in 2010, PyTorch's 2016 release at Facebook AI Research, and the 2022 handover to the PyTorch Foundation

PyTorch is an open-source deep learning library originally built at Facebook AI Research (now Meta AI), and since September 2022 it’s governed independently by the PyTorch Foundation, a subsidiary of the Linux Foundation — Meta still contributes, but no longer owns the project outright. Current stable release is 2.14.0. Its own defining trait, unchanged since 2016, is eager execution: PyTorch runs your code line by line as you write it, the same way plain Python does, rather than asking you to first build a graph and then run it.

That’s the API this guide covers, with facts checked against PyTorch’s own documentation, its GitHub-published history, and Wikipedia’s sourced entry, and code taken directly from PyTorch’s own current quickstart tutorial rather than reconstructed from memory.

PyTorch didn’t start as PyTorch

The lineage most write-ups skip: PyTorch is the third framework in its own family line, not the first.

  1. Torch (2002) — a machine-learning library written in C and Lua at the Idiap Research Institute, by Ronan Collobert and collaborators, supporting neural networks, SVMs, and hidden Markov models.
  2. Torch7 (around 2010) — a rewrite by Collobert, Clément Farabet, and Koray Kavukcuoglu, keeping the C backend but exposing a Lua frontend.
  3. PyTorch (September 2016) — a full break from the Lua frontend, giving Torch’s backend a Python API instead. It was created by Adam Paszke, Sam Gross, Soumith Chintala, and Gregory Chanan, drawing influence from torch-autograd and Chainer’s define-by-run design. Torch7 development stopped in 2018 once PyTorch absorbed its user base.

Two more milestones matter for anyone deciding whether to trust the project long-term. In March 2018, Meta merged Caffe2 — its other, separate deep learning framework, aimed at production inference — into PyTorch, ending the confusing situation where the same company shipped two incompatible frameworks. And in September 2022, Meta handed governance of PyTorch to the newly formed PyTorch Foundation under the Linux Foundation, the same kind of vendor-neutral move Kubernetes made years earlier — the explicit goal being that PyTorch’s roadmap isn’t dictated by one company’s product needs.

Infographic: PyTorch's lineage from the 2002 Torch library, through the 2010 Torch7 rewrite, PyTorch's 2016 release at Facebook AI Research, the 2018 Caffe2 merger, and the 2022 move to the vendor-neutral PyTorch Foundation

PyTorch is licensed under BSD-3, written in Python, C++, and CUDA, and runs on Linux, macOS, and Windows. Beyond NVIDIA’s CUDA, it also supports AMD’s ROCm and Apple’s Metal (as the mps device) — Apple Silicon Macs get real GPU acceleration, not just CPU fallback.

Eager execution, and what that actually means

The single concept that explains most of PyTorch’s design: it’s a define-by-run framework. There’s no separate “build the graph, then run it” step — every line executes immediately, the same way a for loop or a print() statement would, and Python’s own debugger, print statements, and stack traces work on your model exactly as they would on any other Python code.

Under the hood, this is powered by Autograd, PyTorch’s reverse-mode automatic differentiation engine. As your model’s forward pass runs, Autograd records every tensor operation into a directed acyclic graph; when you call .backward(), it walks that graph backward computing gradients. Because the graph is rebuilt fresh on every forward pass, a model can have a different structure on every single call — a different number of loop iterations, a branch taken or not taken — a property that mattered enormously for research code trying variable-length sequences or recursive tree structures, and matters less now that most production code has settled on transformer-shaped models with fixed, regular structure.

Building and training a real model

This is PyTorch’s own current quickstart, unmodified — a full, working training loop on the FashionMNIST dataset, not a toy snippet:

import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets
from torchvision.transforms import v2

training_data = datasets.FashionMNIST(
    root="data", train=True, download=True,
    transform=v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)]),
)
test_data = datasets.FashionMNIST(
    root="data", train=False, download=True,
    transform=v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)]),
)

batch_size = 64
train_dataloader = DataLoader(training_data, batch_size=batch_size)
test_dataloader = DataLoader(test_data, batch_size=batch_size)

A model is a Python class: layers go in __init__, and the forward pass — the part that would be implicit in Keras’s Sequential — is spelled out as ordinary Python in forward():

device = torch.accelerator.current_accelerator().type if torch.accelerator.is_available() else "cpu"

class NeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28*28, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, 10),
        )

    def forward(self, x):
        x = self.flatten(x)
        return self.linear_relu_stack(x)

model = NeuralNetwork().to(device)

The training loop is explicit too — there’s no .fit() call hiding it. This is the part Keras trades away for brevity, and PyTorch trades back for transparency:

loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

def train(dataloader, model, loss_fn, optimizer):
    model.train()
    for X, y in dataloader:
        X, y = X.to(device), y.to(device)
        pred = model(X)
        loss = loss_fn(pred, y)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

for t in range(5):
    train(train_dataloader, model, loss_fn, optimizer)

Every line in that loop is a real, separate step you control: run the model forward, compute the loss, backpropagate, step the optimizer, and zero the gradients before the next batch — nothing is implicit. Saving and reloading is equally direct:

torch.save(model.state_dict(), "model.pth")

model = NeuralNetwork().to(device)
model.load_state_dict(torch.load("model.pth", weights_only=True))

torch.compile: keeping eager mode, adding a compiler underneath

PyTorch 2.0, released 15 March 2023, didn’t change any of the code above — it added an optional, one-line wrapper that compiles it:

model = torch.compile(model)

That single call turns on four new components working together: TorchDynamo captures the Python bytecode of your model safely; AOTAutograd traces the backward pass ahead of time instead of building it fresh every call; PrimTorch reduces PyTorch’s roughly 2,000 operators down to about 250 primitives that a backend actually has to implement; and TorchInductor generates fast GPU code from that reduced set, using OpenAI’s Triton compiler for NVIDIA and AMD hardware.

PyTorch’s own validation ran torch.compile against 163 unmodified open-source models — 46 from Hugging Face Transformers, 61 from Ross Wightman’s TIMM vision models, 56 from TorchBench — and measured, on an NVIDIA A100 GPU:

Infographic: torch.compile's measured results across 163 open-source models on an NVIDIA A100 GPU — working on 93 percent of models, with a 43 percent average training speedup, 21 percent at float32 precision and 51 percent under automatic mixed precision

Sylvain Gugger, the maintainer of Hugging Face Transformers at the time, put it plainly: “With just one line of code to add, PyTorch 2.0 gives a speedup between 1.5x and 2.x in training Transformers models.” The trade PyTorch made deliberately: torch.compile is fully optional and 100% backward compatible — nothing breaks if you never call it, and eager mode, the default since 2016, stays exactly as it was.

The ecosystem around the core library

PyTorch’s own current documentation lists ten actively maintained sibling projects, beyond the two most people already know:

  • torchvision and torchaudio — datasets, pretrained models, and transforms for images and audio, used in the training loop above
  • PyTorch Lightning — a training-loop framework that removes the boilerplate train()/test() functions above without hiding eager execution underneath
  • ExecuTorch — on-device inference for mobile and embedded hardware
  • torchao — quantization and low-precision training utilities
  • TorchRL — building blocks for reinforcement learning
  • torchtitan — reference code for training large models at scale
  • tensordict — a tensor-like container for structured data
  • Helion and kineto — a higher-level compiler DSL and a performance-profiling library, respectively

Hugging Face Transformers, while not a PyTorch-governed project, is built on top of it and is arguably the single biggest reason PyTorch became the default choice in NLP: pretrained model weights for nearly every published architecture, loadable in a couple of lines. OpenAI standardized on PyTorch for its own research in January 2020, and it now underlies ChatGPT, alongside other production systems like Tesla’s Autopilot and Uber’s Pyro probabilistic programming library.

PyTorch vs. Keras and TensorFlow

The honest three-way comparison, after both frameworks have moved since their early reputations formed:

  • Default execution mode. PyTorch has always been eager. TensorFlow was graph-first through version 1.x, switched to eager-by-default in TensorFlow 2.0 (2019), and Keras 3 (2023) runs on top of TensorFlow, JAX, or PyTorch itself — so “PyTorch is dynamic, TensorFlow is static” describes 2017, not 2026.
  • Compilation. Both ecosystems now offer an optional compile step for speed rather than requiring one: PyTorch’s torch.compile versus TensorFlow’s @tf.function and Keras 3’s own multi-backend dispatch. Neither forces you to write your model differently to get it.
  • Where each tends to show up. PyTorch dominates published research and NLP (via Hugging Face); Keras’s pitch is fewer lines of code and running unmodified across three backends; TensorFlow’s long production tooling (TensorFlow Serving, TensorFlow Lite, TensorFlow.js) still gives it an edge in some deployment paths, independent of which framework trained the model.

Model portability has quietly reduced how much this choice matters: ONNX (built by Meta and Microsoft, launched September 2017) exists specifically to move a trained model between these ecosystems, and Keras 3 can already export a model to run as PyTorch, TensorFlow, or JAX regardless of which one trained it.

Frequently asked questions

Who created PyTorch, and when?

Adam Paszke, Sam Gross, Soumith Chintala, and Gregory Chanan, released in September 2016 at Facebook AI Research, as a Python-first successor to the Lua-based Torch7.

Is PyTorch still owned by Meta?

No, not solely. Meta transferred governance to the independent PyTorch Foundation, a Linux Foundation subsidiary, in September 2022. Meta remains a major contributor, not the sole owner.

What does torch.compile actually do?

It’s an optional, one-line wrapper (torch.compile(model)) that compiles your existing eager-mode model into faster GPU code via TorchDynamo, AOTAutograd, PrimTorch, and TorchInductor, without requiring any change to how the model itself is written.

Is PyTorch faster than TensorFlow or Keras?

Neither framework wins outright — Keras’s own published benchmarks show its three backends trading wins depending on the model, and PyTorch’s own torch.compile benchmark reports gains that vary by model family (21% at float32, 51% under mixed precision). Benchmark your actual model rather than trusting a framework-level claim.

What is Torch, and is it different from PyTorch?

Torch was a separate, older (2002) C/Lua library from the Idiap Research Institute. PyTorch (2016) is a from-scratch Python frontend built by different authors, inspired by Torch’s design but not built from its codebase; Torch7 development stopped in 2018.