On 30 November 2020 Hugging Face released transformers 4.0.0, a major release that changes default behaviour in ways that break code written for the 3.x series. For anyone maintaining production pipelines the three most relevant points are: fast tokenizers on by default, return_dict=True as the new standard, and the removal of sentencepiece from the required dependencies. The release notes list them all as explicit breaking changes (release v4.0.0).

Context

transformers is an Apache 2.0 Python library that offers a uniform API over Transformer-architecture models — BERT, RoBERTa, GPT-2, T5, BART and others — with interoperability between PyTorch and TensorFlow 2.0 and experimental Flax support (4.0 documentation). The project started in 2018 as pytorch-pretrained-bert, became pytorch-transformers, and finally took the name transformers once it stopped being limited to PyTorch.

What sets it apart from a plain repository of implementations is weight distribution through the Hub: pretrained weights do not travel with the code but are downloaded at runtime from identifiers such as bert-base-uncased. This ties the library version tightly to the format of published models, and it is why a default change in 4.0 has effects that reach beyond a single project.

The three defaults that change

Fast tokenizers. In 3.x, AutoTokenizer returned the Python (slow) implementation by default. In 4.0 it returns, where one exists, the implementation from the tokenizers library, written in Rust. The release notes recall that the old behaviour comes back with use_fast=False. The difference is not only one of speed: fast and slow tokenizers can produce different offsets and handle edge cases differently, so the silent substitution needs checking on tasks sensitive to token-character alignment, such as named-entity recognition.

return_dict. The model forward pass now returns named-tuple-like objects instead of plain tuples. Code that indexes the output by position — outputs[0] for the logits, say — keeps working, but where strict compatibility is needed return_dict=False brings back 3.x behaviour.

Splitting out sentencepiece. The sentencepiece dependency has left setup.py. The tokenizers for XLNet, ALBERT, CamemBERT, mBART, Pegasus, T5, Reformer and XLM-RoBERTa require it, and it now installs as an extra: pip install transformers[sentencepiece]. An environment that counted on transitive installation breaks on the first import of one of these tokenizers.

On top of all this comes the package reorganisation: the model files have moved into per-architecture subdirectories, and direct imports change from transformers.modeling_bert to transformers.models.bert.modeling_bert. Imports from the top-level namespace (from transformers import BertModel) stay as they were; only code importing internal modules breaks.

The critical point

Hub weight distribution and changing defaults together open a breakage surface that is not obvious. A model published on the Hub carries its tokenizer files but does not pin the library version it will be loaded with. Whoever publishes a checkpoint tested with the slow tokenizer and whoever loads it under 4.0 get, by default, two different tokenizers. Almost always the output coincides; when it does not, the symptom is an accuracy drift that is hard to attribute, because nothing in the code signals that the tokenizer changed.

It is the structural price of an ecosystem where code and weights have separate life cycles and independent versioning. The convenience — a sentiment classifier in three lines — and the risk — behaviour that depends on which version resolves which default — both come from the same decoupling.

from transformers import pipeline
classifier = pipeline("sentiment-analysis")
classifier("open source lends itself to review")
# in 4.0 this loads a fast tokenizer by default

The ecosystem around it

Next to 4.0 sit libraries with their own versioning. tokenizers provides the Rust implementations and is the dependency that enables the fast default. datasets, which reached 1.0 in September 2020, handles dataset loading and streaming, leaning on Apache Arrow so as not to hold everything in memory (datasets repository). Each has its own version number and its own calendar, so compatibility between transformers 4.0 and a specific minor of datasets or tokenizers must be fixed explicitly in the requirements file, not taken for granted.

4.0 also brings new models: mT5 and T5 v1.1, a TensorFlow implementation of DPR, Longformer in TensorFlow with additional task heads. The list of architectures in the 4.0 documentation counts around 34.

Implications for upgraders

The 3.x → 4.0 migration reduces to a few concrete checks. Pin the versions of the four related libraries in the requirements file. Add the [sentencepiece] extra if you use the models that need it. Decide case by case whether the switch to fast tokenizers is safe for your task and, when in doubt, write use_fast=False instead of trusting the default. Update any internal-module imports to the new per-architecture paths.

Under all these points sits the same principle: in a system where defaults change between major versions and weights travel separately from the code, reproducibility comes from explicitly pinning what the library would otherwise decide. A requirements file with the exact versions of transformers, tokenizers, datasets and sentencepiece documents the state in which a result was obtained better than any side note.

Limits

What is written here holds for 4.0.0 and for the behaviour documented in its release notes; later minors may shift other defaults. Flax support is declared experimental and covers only some of the models, so it will not carry a production load. And verifying the divergence between fast and slow tokenizers stays empirical: it depends on the model and the task, and the only reliable way to rule it out is to measure it on your own dataset.

Cover image: Hugging Face logo: a yellow hugging-face emoji with open hands next to the “Hugging Face” wordmark — logo by Victor, Apache 2.0 — https://commons.wikimedia.org/wiki/File:Hf-logo-with-title.svg