Microsoft publishes Turing-NLG and the DeepSpeed library
Microsoft published Turing Natural Language Generation (T-NLG), at 17 billion parameters the largest language model released at the time, reporting state-of-the-art results on several language modeling benchmarks. It was a generative model aimed at practical uses: conversational agents, question answering, and document summarization. What made it possible was released alongside it — the DeepSpeed library and the ZeRO memory optimizer, which cut the degree of model parallelism from 16 to 4, quadrupled batch size per node, and reduced training time threefold, letting training that would have needed 1,024 NVIDIA GPUs under Megatron-LM alone run on 256. The model itself saw little use, but DeepSpeed took hold as open source and became one of the standard tools for large-scale training. It came three months before OpenAI's GPT-3 paper.