Small GPT-2

A minimal PyTorch implementation of a character-level GPT-style language model.

Repository Contents

• main.py - Training script that builds a small Transformer/GPT-like model, trains it on input.txt, saves model.pt, and generates text into output.txt.
• input.txt - Training corpus used by main.py.
• model.pt - Saved model weights after training.
• output.txt - Generated text output produced by the model after training.

What this project does

main.py:

• loads text from input.txt
• builds a character-level vocabulary
• defines a small Transformer-based language model using self-attention blocks
• trains the model with next-character prediction
• saves the trained weights to model.pt
• generates sample text and writes 10,000 generated tokens to output.txt

Requirements

• Python 3.8+
• PyTorch

Install dependencies via pip:

pip install torch

Usage

Run the training and generation script:

python main.py

After completion, the model weights are written to model.pt and generated text is saved to output.txt.

Notes

• The model is character-level, not token-level.
• It uses a small transformer with 6 layers, 6 heads, and a context window of 256 characters.
• Training runs for 5000 iterations by default.