GPT From Scratch

A hands-on mini-project where I built a basic GPT-style character-level language model from scratch using PyTorch. This repo explores how transformers work and how text can be modeled and generated.

What’s Inside

• Tokenization & Preprocessing: Converted raw text into token sequences for model training.
• Bigram Baseline: A simple token-to-next-token predictor to compare against transformer performance.
• Transformer Model: Built transformer blocks featuring multi-head self-attention, feed-forward layers, and layer normalization.
• Text Generation: Trained the model to autoregressively generate text one character at a time.

Tech Stack

• Python
• PyTorch
• NumPy

Setup Instructions

1. Clone this repository:

   git clone https://github.com/Sushitrashhhh/gpt-from-scratch.git
   cd gpt-from-scratch

2. Install dependencies:

   pip install -r requirements.txt