Lazy loaded image8.1 nanoGPT: Training a GPT from Scratch

8.1 nanoGPT: Training a GPT from Scratch

Full code provided in the end for downloading

8.1.1 Preparations

  • Prepare the Environment: We provide two methods for setting up your environment: using Anaconda or Google Colab.
      1. Using Anaconda: Create a new Conda environment and install the required packages. (If you're not familiar with managing Anaconda environments, it's recommended to learn that first.)
          2. ⭐
          Ensure your environment has CUDA version 11.8 or higher
          Run nvidia-smi in the command prompt. The output will include your CUDA version.
          notion image
          • Create the environment:
            • Activate the environment:
              • Install the necessary libraries:
            1. Using CoLab: If you prefer to run the experiment online, click this link: nanoGPT example.
                • You don’t need to manually download the data when using Colab. The first code block fetches the dataset online and loads it into memory:
                  • By default, Colab uses the CPU. To switch to a GPU:
                    • Click Runtime → Change runtime type:
                        • notion image
                    • Select T4 GPU. (Note that TPU v2-8 work only for TensorFlow, but we are using PyTorch.)
                        • notion image
          • Prepare the Data: Download tinyshakespeare dataset used by nanoGPT. You can use one of the following methods:
            • On Linux, use wget:
              • On Windows (Windows 10 or 11), use curl:
                • Or download via your browser. Click the dataset link: tinyshakespeare, then press Ctrl + S to save the file manually. Or, you can copy the text and save it to a .txt file.
              <ins/>

              8.1.2 Code Explanation

              • Import related packages and set hyperparameters
                • Read the dataset, doing a simple tokenization
                    • The output is
                • Dataset splitting and batching
                  • Model Evaluation
                    • Model Definition
                      • A single head in the multi-head self-attention mechanism
                        • Multi-head self-attention, FFN, and block definition:
                          • Define a Bigram Language Model
                          • Model Training
                            <ins/>

                            8.1.3 Experimental Results

                            • Experimental Results: First, let's examine the model parameters. From the implementation above, we can see that this small GPT-2 model has approximately 0.209729 million parameters. Using the rough estimation method introduced earlier:
                                • This estimate is roughly consistent with the actual value.
                            • Generate new text from a blank context:
                              • Training loss
                                  • notion image
                              • GPU Memory Usage During Training
                                • Training requires only 348MB of GPU memory. The code uses FP32 precision, meaning each element occupies 4 bytes.
                                    • notion image
                                • When CUDA is initialized, it consumes 298MB for context initialization. You can observe this by running torch.tensor(1).to('cuda'), which triggers CUDA initialization in PyTorch.
                                    • notion image
                                • The remaining memory is used for model parameters, gradients, optimizer state, intermediate activations, and the PyTorch memory cache.
                                    • To understand how much memory is consumed by intermediate activations, it's helpful to print torch.cuda.memory_allocated() after each layer during a forward pass.
                                    ⭐
                                    Monitor GPU usage.
                                    Instead of using nvidia-smi for a one-time snapshot of GPU usage, you can use these Python-based tools:
                                    1. gpustat: Install with pip install gpustat. Run it in an activated environment to get a detailed overview of GPU usage.
                                    1. nvitop: Install with pip install nvitop. Run it to get a dynamic, real-time view of GPU processes and memory usage.
                                    ⚠️ On Windows, neither gpustat nor nvitop can provide detailed per-process GPU memory usage.
                                Prev
                                7.3 Knowledge Distillation
                                Next
                                8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
                                Loading...
                                Article List
                                LLM Learning Roadmap
                                ✨ Awesome-Anything
                                Lazy loaded image Awesome-AI-Tutorials
                                Lazy loaded image Awesome-AI-Researchs
                                Lazy loaded image Awesome-GitHub-Repos
                                Lazy loaded image Awesome-AI-Examples
                                🖼️ Digital Image Processing
                                Lazy loaded image DIP I Image Demosaicing and Histogram Manipulation
                                Lazy loaded image DIP II Image Denoising
                                Lazy loaded image DIP III Edge Detection
                                Lazy loaded image DIP IV Digital Half-toning
                                Lazy loaded image DIP V Geometric Image Modification
                                🍃 LLM Components
                                Lazy loaded image 1.1 Tokenization
                                Lazy loaded image 1.2 Embedding
                                Lazy loaded image 1.3 Attention
                                Lazy loaded image 1.4 FFN, Residual Addition, and LN
                                Lazy loaded image 1.5 Positional Encoding
                                Lazy loaded image 1.6 Architecture & Decoding Policy
                                🌱 LLM Pre-training
                                Lazy loaded image 2.1 Pre-training Definition
                                Lazy loaded image 2.2 Pre-training Data
                                Lazy loaded image 2.3 Pre-training Workflow
                                Lazy loaded image 2.4 Continued Pre-training (CPT)
                                Lazy loaded image 2.5 Pre-training Evaluation
                                ☘️ LLM Post-Training
                                Lazy loaded image 3.1 Supervised Fine-Tuning (SFT)
                                Lazy loaded image 3.2 Reinforcement Learning
                                Lazy loaded image 3.3 RLHF: Reinforcement Learning from Human Feedback
                                Lazy loaded image 3.4 Optimizing PPO-Based Algorithms
                                Lazy loaded image 3.5 Direct Preference Optimization (DPO)
                                🍀 LLM Popular Models
                                Lazy loaded image 4.1 BERT & Variants
                                Lazy loaded image 4.2 PaLM Series
                                Lazy loaded image 4.3 GPT Series
                                Lazy loaded image 4.4 LLaMA Series
                                Lazy loaded image 4.5 GLM Series
                                Lazy loaded image 4.6 Qwen Series
                                Lazy loaded image 4.7 DeepSeek Series
                                Lazy loaded image 4.8 MOE Series
                                Lazy loaded image 4.9 Other
                                🪴 LLM Applications
                                Lazy loaded image 5.1 Prompt Engineering
                                Lazy loaded image 5.2 LLM-based Agents
                                Lazy loaded image 5.3 Retrieval-Augmented Generation (RAG)
                                Lazy loaded image 5.4 Enterprise-level RAG Practice (I)
                                Lazy loaded image 5.5 Enterprise-level RAG Practice (II)
                                🌿 LLM Optimization
                                Lazy loaded image 6.1 GPU Usage Analysis
                                Lazy loaded image 6.2 Latency Analysis for Inference
                                Lazy loaded image 6.3 FlashAttention
                                Lazy loaded image 6.4 PagedAttention
                                Lazy loaded image 6.5 Packing for Efficient Training
                                Lazy loaded image 6.6 PEFT
                                🌾 LLM Compression
                                Lazy loaded image 7.1 Quantization
                                Lazy loaded image 7.2 Pruning
                                Lazy loaded image 7.3 Knowledge Distillation
                                🌵 LLM Hands-on Practice
                                Lazy loaded image 8.1 nanoGPT: Training a GPT from Scratch
                                Lazy loaded image 8.2 Instruction Fine-Tuning Qwen3-0.6B: A Minimal Working Example
                                Lazy loaded image 8.3 OpenRLHF: SFT Qwen3-0.6B and Code Analysis
                                Lazy loaded image 8.4 OpenRLHF: Reward Model Training and Code Analysis
                                Lazy loaded image 8.5 OpenRLHF: PPO Tutorial and Code Analysis
                                🌴 LLM Must-read Papers
                                Lazy loaded image 9.1 Technical Reports Explained
                                Lazy loaded image 9.2 MedLLM
                                Lazy loaded image 9.3 o1 Techinical Paper
                                Lazy loaded image 9.4 Prompting
                                Lazy loaded image 9.5 LLM-as-Judge
                                🌳 LLM Q&A
                                Lazy loaded image 10.1 Why GRPO training loss starts at 0
                                Lazy loaded image 10.2 VLM-R1 Reproduction Results
                                Lazy loaded image 10.3 KL(P||Q) or KL(Q||P)?
                                Lazy loaded image 10.4 Left Padding or Right Padding?
                                Lazy loaded image 10.5 What is QK-Norm?
                                Lazy loaded image 10.6 Why LLM training rarely adopts Dropout?
                                Lazy loaded image 10.7 Dropout and DropPath
                                🐝 VLM Image Encoders
                                Lazy loaded image 11.0 Introduction
                                Lazy loaded image 11.1 Vision Transformer
                                Lazy loaded image 11.2 CLIP
                                Lazy loaded image 11.3 BLIP
                                Lazy loaded image 11.4 EVA
                                Lazy loaded image 11.6 BLIP-2
                                Lazy loaded image 11.7 SigLIP
                                📝 MISC.
                                🤔 Large Language Models Pass the Turing Test.
                                Lazy loaded image OpenAI ChatGPT Progression
                                ⏳ Claude Sonnet 4 Test with Sandtris
                                🔍 What DeepSeek-R1's Surge Teaches Us About SEO
                                💰 CivitAI’s Payment Issue
                                🎓 arXivSub, an Academia Assistant