GPT-2 Full rewrite

This repository contains a full rewrite of the GPT-2 model with many layers and concepts implemented from scratch. It is a learning resource for understanding the inner workings of the model.

Table of contents:

• Concepts - Jupyter notebooks with visualizations and explanations of the concepts

  • Positional encoding +
  • Backpropagation
  • Cross-entropy
  • Optimizers

• Layers - Implementations of the layers used in the model

  • Embedding
  • Transformer
  • Linear
  • Normalization
  • Optimizations
    • RoPE Implementation
    • KV Cache Implementation

• Model - The main model implementation

• Train - Training loop implementation

• Checkpoints - Checkpoint for the model