Build a Large Language Model (From Scratch)

Learn how to create, train, and tweak large language models (LLMs) by building one from the ground up!In Build a Large Language Model (from Scratch) bestselling author Sebastian Raschka guides you step by step through creating your own LLM. Each stage is explained with clear text, diagrams, and examples. You’ll go from the initial design and creation, to pretraining on a general corpus, and on to fine-tuning for specific tasks.Build a Large Language Model (from Scratch) teaches you how to:Plan and code all the parts of an LLMPrepare a dataset suitable for LLM trainingFine-tune LLMs for text classification and with your own dataUse human feedback to ensure your LLM follows instructionsLoad pretrained weights into an LLMBuild a Large Language Model (from Scratch) takes you inside the AI black box to tinker with the internal systems that power generative AI. As you work through each key stage of LLM creation, you’ll develop an in-depth understanding of how LLMs work, their limitations, and their customization methods. Your LLM can be developed on an ordinary laptop, and used as your own personal assistant.About the technologyPhysicist Richard P. Feynman reportedly said, “I don’t understand anything I can’t build.” Based on this same powerful principle, bestselling author Sebastian Raschka guides you step by step as you build a GPT-style LLM that you can run on your laptop. This is an engaging book that covers each stage of the process, from planning and coding to training and fine-tuning.About the bookBuild a Large Language Model (From Scratch) is a practical and eminently-satisfying hands-on journey into the foundations of generative AI. Without relying on any existing LLM libraries, you’ll code a base model, evolve it into a text classifier, and ultimately create a chatbot that can follow your conversational instructions. And you’ll really understand it because you built it yourself!What's insidePlan and code an LLM comparable to GPT-2Load pretrained weightsConstruct a complete training pipelineFine-tune your LLM for text classificationDevelop LLMs that follow human instructionsAbout the readerReaders need intermediate Python skills and some knowledge of machine learning. The LLM you create will run on any modern laptop and can optionally utilize GPUs.About the AuthorSebastian Raschka has been working on machine learning and AI for more than a decade. Sebastian joined Lightning AI in 2022, where he now focuses on AI and LLM research, developing open-source software, and creating educational material. Prior to that, Sebastian worked at the University of Wisconsin-Madison as an assistant professor in the Department of Statistics, focusing on deep learning and machine learning research. He has a strong passion for education and is best known for his bestselling books on machine learning using open-source software. Table of Contents1 Understanding large language models2 Working with text data3 Coding attention mechanisms4 Implementing a GPT model from scratch to generate text5 Pretraining on unlabeled data6 Fine-tuning for classification7 Fine-tuning to follow instructionsA Introduction to PyTorchB References and further readingC Exercise solutionsD Adding bells and whistles to the training loopE Parameter-efficient fine-tuning with LoRA