A BERT model pre-trained on PubMed abstracts is not the best starting point for reading a discharge letter: scientific literature and ward notes have different lexical and syntactic distributions, and that difference shows in the downstream task metrics. This is why, between 2019 and 2022, biomedical encoders branched into several lines instead of converging on a single model.

Context

BERT (Bidirectional Encoder Representations from Transformers), published by Devlin et al. in 2018, defines a two-stage scheme: self-supervised pre-training on unlabelled text via masked language modelling, then supervised fine-tuning on a specific task. BERT-base is pre-trained on English Wikipedia and BookCorpus, general-domain text. For the biomedical domain this is a concrete limit: terms such as levothyroxine, BRCA1 or dyspnoea occur rarely in the general corpus, and the WordPiece tokenizer splits them into many sub-tokens, losing the morphological structure that makes them informative.

The biomedical NLP (natural language processing) community did not answer with a single model. Three pre-training choices, distinct and partly alternative, set the picture as of 2022.

Three pre-training strategies

BioBERT (Lee et al., Bioinformatics 36(4), 2020) starts from BERT-base weights and continues pre-training on 4.5 billion words of PubMed abstracts and 13.5 billion words of PMC (PubMed Central) full text. The point is that it inherits BERT’s original WordPiece vocabulary: the model sees biomedical text but keeps tokenising with a vocabulary tuned on general text. On biomedical named entity recognition (NER) benchmarks — NCBI Disease, BC5CDR, diseases and chemical compounds — BioBERT gains a few F1 points over BERT-base, with improvements also on relation extraction and BioASQ question answering. The weights ship under the Apache 2.0 licence; the most used variant is dmis-lab/biobert-base-cased-v1.1, where the cased version preserves capitalisation: CD4 and cd4 are not the same thing.

ClinicalBERT (Alsentzer et al., Clinical NLP Workshop, NAACL 2019) instead starts from BioBERT and continues pre-training on MIMIC-III, the public intensive-care note dataset from Beth Israel Deaconess Medical Center. Here the register shifts: clinical text is telegraphic, dense with local abbreviations (pt, s/p, c/o), often without full syntax. The resulting model, distributed as emilyalsentzer/Bio_ClinicalBERT, does better on tasks built on ward notes (the i2b2/n2c2 challenges) than an encoder trained on literature alone.

PubMedBERT (Gu et al., 2020; later ACM Transactions on Computing for Healthcare, 2021) overturns a widespread assumption. Rather than continuing pre-training from BERT, it trains from scratch on PubMed and — here is the point — builds the WordPiece vocabulary directly from the biomedical corpus. The model sees 14 million abstracts (about 3 billion words, 21 GB) and tokenises medical terms more compactly. The same work introduces BLURB (Biomedical Language Understanding and Reasoning Benchmark), aggregating NER, PICO extraction, relation extraction, sentence similarity, classification and QA. On BLURB, PubMedBERT beats BioBERT on most tasks. It is distributed under the MIT licence.

The critical point: vocabulary and corpus provenance

The three choices isolate two independent variables. The first is corpus provenance: literature (PubMed/PMC) or clinical notes (MIMIC). The second is the tokenisation vocabulary: inherited from general-domain BERT or rebuilt from the domain.

BioBERT changes only the corpus and keeps the general vocabulary. PubMedBERT changes both. ClinicalBERT changes the corpus again (towards the clinical side) but inherits the vocabulary from BioBERT. The Gu et al. result — rebuilding the vocabulary from the corpus yields measurable gains on a homogeneous benchmark — isolates the effect of tokenisation on performance. A term such as lymphangioleiomyomatosis, which a general vocabulary fragments into many pieces, becomes a shorter, more coherent sequence under a PubMed vocabulary.

This is why there is no single dominant encoder: the choice depends on the task. For NER on scientific abstracts, PubMedBERT is a solid reference. For de-identification or coding on discharge letters, a model exposed to real clinical text such as ClinicalBERT does better. The provenance of the evaluation text must match that of the pre-training text.

Scale as a separate variable

A separate line measures what changes with size. GatorTron (Yang et al., medRxiv/arXiv 2203.03540, February–March 2022, University of Florida with NVIDIA) trains from scratch on over 90 billion words, of which more than 82 billion come from de-identified UF Health clinical notes, plus literature and general text. It releases three variants: 345 million, 3.9 billion and 8.9 billion parameters, against the ~110 million of BERT-base. The models up to 3.9 billion are published on NVIDIA’s NGC catalogue.

GatorTron shows that, with the encoder architecture held fixed, more parameters and more corpus improve five clinical NLP tasks. It is a result about scale, orthogonal to the corpus/vocabulary choice above: it changes the order of magnitude of the resources, not the nature of the model, which remains an encoder-only transformer and produces contextual representations.

Infrastructure

All these models share one entry point: the Hugging Face Transformers library and the matching Hub. Loading a biomedical encoder is uniform:

from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("dmis-lab/biobert-base-cased-v1.1")
model = AutoModel.from_pretrained("dmis-lab/biobert-base-cased-v1.1")

Changing the identifier to emilyalsentzer/Bio_ClinicalBERT or to the PubMedBERT variant changes the model without touching the rest of the pipeline. It is this interface uniformity that makes pre-training choices comparable: the same classification head, the same fine-tuning loop, different encoders.

Limits

As of June 2022 the constraints are explicit.

Language: almost all of these models are trained on English. For Italian clinical text there is no reference pre-trained encoder of comparable maturity; application requires either translation or pre-training on Italian corpora that are not yet widely available.

Context length: 512 tokens in the base variants. A full discharge letter often exceeds this limit, forcing segmentation and the loss of long-range dependencies. Architectures such as Longformer or BigBird widen the window but are not yet standard in the clinical domain.

Reasoning: an encoder produces representations and classifications, not inference chains. Tasks that ask to chain several facts remain outside this scheme.

Regulated use: integrating one of these models into software as a medical device requires qualification under IEC 62304, risk management and clinical validation. Availability under Apache 2.0 or MIT enables experimentation, not clinical deployment without the conformity path.

The first reference to fix remains the match between pre-training corpus and application domain: it is the variable that, before scale and before fine-tuning, decides whether a biomedical encoder is the right starting point.

https://academic.oup.com/bioinformatics/article/36/4/1234/5566506 https://github.com/dmis-lab/biobert https://arxiv.org/abs/1904.03323 https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT https://arxiv.org/abs/2007.15779 https://arxiv.org/abs/2203.03540 https://huggingface.co/docs/transformers/index https://www.noze.it/en/insights/biobert-clinical-pubmed-transformers/

Cover image: Block diagram of the Transformer model architecture: encoder column on the left and decoder column on the right, with multi-head… — diagram by Yuening Jia, CC BY-SA 3.0 — https://commons.wikimedia.org/wiki/File:The-Transformer-model-architecture.png