"Textbooks Are All You Need" — Microsoft's tiny phi-1 model

Microsoft Research published "Textbooks Are All You Need," presenting phi-1, a small model for code generation. At just 1.3 billion parameters and four days of training on eight A100 GPUs, it reached pass@1 accuracy of 50.6% on HumanEval and 55.5% on MBPP. The key was training on carefully curated "textbook quality" data — textbook-grade prose plus exercises generated with GPT-3.5 — instead of a large mass of miscellaneous web text. Showing that data quality could substitute for scale gave the small language model movement its footing. Phi-2, at 2.7 billion parameters, followed in December, said to match or beat models up to 25 times its size on complex benchmarks. For Microsoft it was the start of a two-track approach: depend on OpenAI's large models while cultivating its own line of small ones.