January

2 events

Editors' summary

Getting bigger becomes an equation

On 23 January, OpenAI published its scaling laws paper, showing that loss falls smoothly as model size, data and compute grow. It became possible to read off how much a given investment would buy, and scale itself became a strategy.

On the 28th, Google announced Meena, a 2.6-billion-parameter dialogue model aimed at talking about anything, along with its own metric for how human a conversation felt. It was not released publicly.

This block is written by the editors. It is kept separate from the sourced record below.

Record2 events

23
ResearchOpenAI / Jared Kaplan / Sam McCandlish / Dario Amodei
Scaling Laws for Neural Language Models paper published
OpenAI researchers including Jared Kaplan and Sam McCandlish published "Scaling Laws for Neural Language Models" on arXiv. The paper showed empirically that language model loss improves as a power law with model size, dataset size, and compute, holding across more than seven orders of magnitude. It also found that larger models are more sample-efficient, giving quantitative grounding to the "bigger is better" intuition. These findings underpinned the scaling strategy behind GPT-3 later that year.
28
ResearchGoogle / Quoc Le / Daniel Adiwardana
Google presents Meena, a chatbot meant to discuss anything
Google presented Meena, a 2.6-billion-parameter conversational model trained end to end for open-domain dialogue rather than any particular topic, reporting that it held conversations more sensible and specific than the chatbots that were then state of the art. To measure it, the team introduced a new metric, Sensibleness and Specificity Average (SSA), capturing basic but important properties of human conversation. The work fed into LaMDA the following year. It shows Google had conversational AI as a research result almost three years before ChatGPT — and did not ship it as a product.

↑ 2020

Scaling Laws for Neural Language Models paper published

Google presents Meena, a chatbot meant to discuss anything