BERT makes pre-training the standard in language processing

Google researchers Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova published "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Using the Transformer's encoder, it pre-trained representations from unlabeled text by conditioning jointly on both left and right context, pushing GLUE to 80.5% — a 7.7 point gain — and SQuAD v1.1 F1 to 93.2. Google open-sourced the model and code the following month. The paper established pre-train-on-lots-of-text then fine-tune-on-a-little as the standard recipe in natural language processing. BERT used only the Transformer's encoder — built for reading text rather than writing it — a different design from the decoder-based GPT line.