Google announces PaLM, a 540-billion-parameter model
Google announced the Pathways Language Model (PaLM), a 540-billion-parameter decoder-only Transformer trained on 780 billion tokens of high-quality text using Pathways, a system for efficiently training a single model across multiple TPU v4 pods. Evaluated on hundreds of language understanding and generation tasks, it set new few-shot state of the art on most of them, often by wide margins, and reached 57.8% hardware FLOPs utilization — the highest for a model of that scale at the time. It also demonstrated large gains on reasoning tasks from chain-of-thought prompting. As research it was at the frontier, but it was never released as a consumer product; its successor, PaLM 2, would be used as the basis for Bard in May 2023.