2020
From scaling laws to GPT-3, research turns into a product
In January, OpenAI published its scaling laws paper, showing in equations how performance improves smoothly as model size, data and compute increase. Where to put resources became something you could read off a curve.
May brought GPT-3, the same idea carried further: 175 billion parameters, taking on new tasks from a handful of examples without further training. In June, OpenAI launched its first commercial product, the OpenAI API, invitation-only at first, and text generation began appearing inside other companies' products. In September, Microsoft took an exclusive licence to GPT-3; back in May it had announced a supercomputer on Azure built for OpenAI.
In November, DeepMind's AlphaFold2 reached near-experimental accuracy in protein structure prediction, hailed as solving a fifty-year-old problem. Google had also announced the Meena dialogue model in January.
In February, Microsoft released Turing-NLG and DeepSpeed. At 17 billion parameters, it was the largest language model at the time.
People moved at the end of the year. On 2 December, Timnit Gebru, co-lead of Google's Ethical AI team, left the company, with the two sides disagreeing over whether she was fired or resigned. On the 29th, Dario Amodei and others leading research at OpenAI departed.
This block is written by the editors. It is kept separate from the sourced record below.
- January2Getting bigger becomes an equation
- February1Microsoft ships the largest model yet, and the tooling to run it
- May2GPT-3 takes on tasks from a handful of examples
- June1OpenAI gets something to sell, and starts earning from it
- September1Microsoft takes an exclusive licence to GPT-3
- November1AlphaFold2 reads the shape of a protein
- December2People working on ethics and safety leave, one after another