Table of Contents

1. Overview
2. Architecture
   • 2.1 Model Architecture
   • 2.2 Transformer Block Structure
3. Theoretical Background
   • 3.1 Knowledge Tracing
   • 3.2 Attentive Knowledge Tracing
   • 3.3 Model Variants
4. System Architecture
5. Data Flow
6. Prerequisites
7. Installation
   • 7.1 Clone the Repository
   • 7.2 Install Dependencies
   • 7.3 Verify Installation
8. Dataset Format
   • 8.1 Standard Format (DATA)
   • 8.2 PID Format (PID_DATA)
   • 8.3 Supported Datasets
9. Configuration
   • 9.1 Command-Line Arguments
   • 9.2 Default Values
10. Usage
    • 10.1 Basic Usage
    • 10.2 Custom Configuration
    • 10.3 Training Process
    • 10.4 Output Files
11. Model Variants
    • 11.1 AKT-PID (Rasch Model)
    • 11.2 AKT-CID (Non-Rasch Model)
12. Training Details
    • 12.1 Training Loop
    • 12.2 Loss Function
    • 12.3 Evaluation Metrics
13. Technical Implementation Details
    • 13.1 Attention Mechanism
    • 13.2 Data Processing
    • 13.3 Model Saving
14. Troubleshooting
    • 14.1 Common Issues
    • 14.2 Reproducibility

1. Overview

This project provides a PyTorch implementation of Attentive Knowledge Tracing (AKT), a state-of-the-art approach for modeling student learning and predicting their performance on educational tasks. AKT leverages transformer-based attention mechanisms to capture the complex relationships between questions, answers, and student knowledge states over time.

The implementation supports two main variants:

• AKT-PID (Rasch Model): Incorporates Problem ID embeddings to model question difficulty
• AKT-CID (Non-Rasch Model): Uses Concept ID without explicit difficulty modeling

Folder PATH listing
+---assets                      <-- Project assets
│       banner.png              <-- Project banner image
│
+---data                        <-- Dataset directories
│   +---assist2009_pid          <-- ASSISTments 2009 dataset with PID
│   │
│   +---assist2015              <-- ASSISTments 2015 dataset
│   │
│   +---assist2017_pid          <-- ASSISTments 2017 dataset with PID
│   │
│   +---statics                 <-- Statics dataset
│
+---data_loaders                <-- Data loading module
│       load_data.py            <-- Data loading classes
│
+---docs                        <-- Documentation files
│       API.md                  <-- API documentation
│
+---model                       <-- Saved model checkpoints
│   +---akt_pid                 <-- AKT-PID model directory
│
+---models                      <-- Model implementations
│       akt.py                  <-- AKT model implementation
│
+---result                      <-- Training results
│   +---akt_pid                 <-- AKT-PID results
│
+---scripts                     <-- Entry point scripts
│       main.py                 <-- Main entry point
│
+---training                    <-- Training module
│       run.py                  <-- Training and testing functions
│
+---utils                       <-- Utility functions
│       utils.py                <-- Utility functions
│
    .gitignore                  <-- Git exclusions
    LICENSE                     <-- License file
    pyproject.toml              <-- Project configuration
    README.md                   <-- Project documentation
    requirements.txt            <-- Python dependencies

2. Architecture

2.1 Model Architecture

The AKT model is built on a transformer architecture that processes sequences of student interactions. The model consists of several key components:

1. Embedding Layers: Convert question IDs and question-answer pairs into dense vector representations
2. Transformer Blocks: Two sets of transformer blocks that encode question-answer sequences and question sequences
3. Multi-Head Attention: Captures temporal dependencies and relationships between interactions
4. Output Layers: Predicts the probability of correct answers

graph TB
    subgraph Input["Input Data"]
        Q[Question IDs]
        QA[Question-Answer IDs]
        PID[Problem IDs<br/>Optional]
    end
    
    subgraph Embeddings["Embedding Layers"]
        QEmb[Question Embedding]
        QAEmb[QA Embedding]
        PIDEmb[PID Difficulty Embedding<br/>if PID model]
    end
    
    subgraph Encoder1["QA Encoder Blocks"]
        Block1_1[Transformer Block 1]
        Block1_2[Transformer Block 2]
        Block1_N[Transformer Block N]
    end
    
    subgraph Encoder2["Question Encoder Blocks"]
        Block2_1[Transformer Block 1<br/>Peek Current]
        Block2_2[Transformer Block 2<br/>No Peek]
        Block2_3[Transformer Block 3<br/>Peek Current]
        Block2_4[Transformer Block 4<br/>No Peek]
        Block2_N[Transformer Block 2N]
    end
    
    subgraph Output["Output Layer"]
        Concat[Concatenate<br/>Encoded Q + Q Embedding]
        FC1[FC: d_model+embed_l → 512]
        FC2[FC: 512 → 256]
        FC3[FC: 256 → 1]
        Sigmoid[Sigmoid]
        Pred[Prediction]
    end
    
    Q --> QEmb
    QA --> QAEmb
    PID -.->|if PID model| PIDEmb
    
    QEmb --> PIDEmb
    PIDEmb --> QEmb
    QAEmb --> PIDEmb
    PIDEmb --> QAEmb
    
    QAEmb --> Encoder1
    QEmb --> Encoder2
    
    Encoder1 --> Block1_1
    Block1_1 --> Block1_2
    Block1_2 --> Block1_N
    
    Encoder2 --> Block2_1
    Block2_1 --> Block2_2
    Block2_2 --> Block2_3
    Block2_3 --> Block2_4
    Block2_4 --> Block2_N
    
    Block1_N -->|"Encoded QA (y)"| Block2_2
    Block1_N -->|"Encoded QA (y)"| Block2_4
    
    Block2_N --> Concat
    QEmb --> Concat
    Concat --> FC1
    FC1 --> FC2
    FC2 --> FC3
    FC3 --> Sigmoid
    Sigmoid --> Pred

Loading

2.2 Transformer Block Structure

Each transformer block contains:

• Multi-Head Attention: With positional effects and masking
• Layer Normalization: Applied after attention and feedforward
• Feedforward Network: Two linear layers with ReLU activation
• Residual Connections: Around both attention and feedforward layers

graph LR
    Input[Input] --> Attn[Multi-Head Attention]
    Attn --> Add1[Add & Norm]
    Input --> Add1
    Add1 --> FFN[Feedforward Network]
    FFN --> Add2[Add & Norm]
    Add1 --> Add2
    Add2 --> Output[Output]
    
    subgraph FFN_Detail["Feedforward Details"]
        Linear1[Linear: d_model → d_ff]
        ReLU[ReLU]
        Dropout1[Dropout]
        Linear2[Linear: d_ff → d_model]
        Dropout2[Dropout]
    end
    
    FFN -.-> FFN_Detail

Loading

3. Theoretical Background

3.1 Knowledge Tracing

Knowledge Tracing is the task of modeling a student's knowledge state over time based on their interaction history. The goal is to predict whether a student will answer a question correctly, given their past performance.

3.2 Attentive Knowledge Tracing

AKT extends traditional knowledge tracing by:

• Using attention mechanisms to weight the importance of past interactions
• Incorporating positional effects to model temporal decay
• Supporting problem difficulty modeling through PID embeddings (Rasch variant)
• Leveraging transformer architecture for better sequence modeling

3.3 Model Variants

AKT-PID (Rasch Model):

• Includes Problem ID embeddings that learn difficulty parameters
• Applies L2 regularization on difficulty parameters
• Suitable for datasets with problem-level annotations

AKT-CID (Non-Rasch Model):

• Uses only question and answer information
• Simpler architecture without difficulty modeling
• Suitable for datasets without problem-level metadata

4. System Architecture

The system is organized into modular components that handle different aspects of the training and evaluation pipeline:

graph TB
    subgraph Entry["Entry Point"]
        Main[scripts/main.py]
    end
    
    subgraph Config["Configuration"]
        Args[parse_args]
        ConfigClass[AKTConfig]
        Constants[constants.py]
    end
    
    subgraph Data["Data Loading"]
        DataLoader[DATA / PID_DATA]
        LoadData[load_data]
    end
    
    subgraph Model["Model"]
        ModelUtils[model_utils.py]
        LoadModel[load_model]
        AKTModel[AKT Model]
    end
    
    subgraph Training["Training"]
        Trainer[trainer.py]
        TrainFunc[train]
        TestFunc[test]
    end
    
    subgraph Utils["Utilities"]
        Logging[logging.py]
        ModelUtils2[model_utils.py]
    end
    
    Main --> Args
    Args --> ConfigClass
    ConfigClass --> Constants
    Main --> DataLoader
    DataLoader --> LoadData
    Main --> ModelUtils
    ModelUtils --> LoadModel
    LoadModel --> AKTModel
    Main --> Trainer
    Trainer --> TrainFunc
    Trainer --> TestFunc
    TrainFunc --> AKTModel
    TestFunc --> AKTModel
    Main --> Logging
    Main --> ModelUtils2

Loading

5. Data Flow

The data processing pipeline transforms raw CSV files into model inputs:

flowchart LR
    subgraph Input["Input Files"]
        CSV[CSV Files<br/>train/valid/test]
    end
    
    subgraph Processing["Data Processing"]
        Load[Load CSV]
        Parse[Parse Lines]
        Split[Split Sequences<br/>if > seqlen]
        Pad[Zero Padding]
        Array[NumPy Arrays]
    end
    
    subgraph Training["Training Loop"]
        Batch[Create Batches]
        Transpose[Transpose Data]
        Shuffle[Shuffle]
        Tensor[Convert to Tensors]
    end
    
    subgraph Model["Model Forward"]
        Embed[Embeddings]
        Encode[Transformer Encoding]
        Predict[Predict]
    end
    
    subgraph Output["Output"]
        Loss[Compute Loss]
        Metrics[AUC/Accuracy]
        Save[Save Model/Results]
    end
    
    CSV --> Load
    Load --> Parse
    Parse --> Split
    Split --> Pad
    Pad --> Array
    Array --> Batch
    Batch --> Transpose
    Transpose --> Shuffle
    Shuffle --> Tensor
    Tensor --> Embed
    Embed --> Encode
    Encode --> Predict
    Predict --> Loss
    Loss --> Metrics
    Metrics --> Save

Loading

6. Prerequisites

The following environment is required to run this project:

• Operating System: Linux (recommended) or compatible Unix-like system
• Python: 3.7 or higher
• PyTorch: 1.2.0 or higher
• NumPy: 1.17.2 or higher
• Scikit-learn: 0.21.3 or higher
• SciPy: 1.3.1 or higher

7. Installation

7.1 Clone the Repository

git clone <repository-url>
cd Interactive-Proficiency-AKT

7.2 Install Dependencies

Using pip:

pip install -r requirements.txt

Or using the project's pyproject.toml:

pip install -e .

For development dependencies:

pip install -e ".[dev]"

7.3 Verify Installation

Ensure PyTorch is properly installed and can detect your hardware:

python -c "import torch; print(torch.__version__); print('CUDA available:', torch.cuda.is_available())"

8. Dataset Format

The project supports two data formats depending on whether Problem IDs are available:

8.1 Standard Format (DATA)

For datasets without Problem IDs (e.g., assist2015, statics):

Each student's data is represented by 3 consecutive lines:

• Line 0: Student ID (not used in processing)
• Line 1: Question sequence (comma-separated question IDs)
• Line 2: Answer sequence (comma-separated: 0 for incorrect, 1 for correct)

Example:

student_0
1,2,3,4,5
1,0,1,1,0

8.2 PID Format (PID_DATA)

For datasets with Problem IDs (e.g., assist2009_pid, assist2017_pid):

Each student's data is represented by 4 consecutive lines:

• Line 0: Student ID (not used in processing)
• Line 1: Problem ID sequence (comma-separated problem IDs)
• Line 2: Question sequence (comma-separated question IDs)
• Line 3: Answer sequence (comma-separated: 0 for incorrect, 1 for correct)

Example:

student_0
101,102,103,104,105
1,2,3,4,5
1,0,1,1,0

8.3 Supported Datasets

The following datasets are pre-configured:

Dataset	Questions	Problem IDs	Format	Model Variant
assist2009_pid	110	16,891	PID	akt_pid
assist2017_pid	102	3,162	PID	akt_pid
assist2015	100	N/A	Standard	akt_cid
statics	1,223	N/A	Standard	akt_cid

9. Configuration

9.1 Command-Line Arguments

The model can be configured through command-line arguments. Key parameters include:

Basic Parameters

• --max_iter: Maximum number of training iterations (default: 300)
• --train_set: Training set fold number (default: 1)
• --seed: Random seed for reproducibility (default: 224)

Model Parameters

• --model: Model type - akt_pid or akt_cid (default: akt_pid)
• --dataset: Dataset name (default: assist2009_pid)
• --d_model: Transformer model dimension (default: 256)
• --d_ff: Feedforward network dimension (default: 1024)
• --n_block: Number of transformer blocks (default: 1)
• --n_head: Number of attention heads (default: 8)
• --dropout: Dropout rate (default: 0.05)
• --kq_same: Whether to use same key and query (1) or different (0) (default: 1)
• --l2: L2 regularization penalty for difficulty parameter (default: 1e-5)

Training Parameters

• --batch_size: Batch size (default: 24)
• --lr: Learning rate (default: 1e-5)
• --maxgradnorm: Maximum gradient norm for clipping (default: -1.0, disabled)
• --final_fc_dim: Final fully connected layer dimension (default: 512)

9.2 Default Values

Default values are defined in src/akt/constants.py:

• DEFAULT_BATCH_SIZE: 24
• DEFAULT_SEQ_LEN: 200
• DEFAULT_LEARNING_RATE: 1e-5
• DEFAULT_MAX_ITER: 300
• DEFAULT_SEED: 224
• DEFAULT_DROPOUT: 0.05
• DEFAULT_D_MODEL: 256
• DEFAULT_D_FF: 1024
• DEFAULT_N_BLOCK: 1
• DEFAULT_N_HEAD: 8
• DEFAULT_L2: 1e-5
• EARLY_STOPPING_PATIENCE: 40

10. Usage

10.1 Basic Usage

Train and test the AKT-PID model on ASSISTments2009:

python scripts/main.py --dataset assist2009_pid --model akt_pid

Train and test the AKT-PID model on ASSISTments2017:

python scripts/main.py --dataset assist2017_pid --model akt_pid

Train and test the AKT-CID model on ASSISTments2015:

python scripts/main.py --dataset assist2015 --model akt_cid

10.2 Custom Configuration

Override default parameters:

python scripts/main.py \
    --dataset assist2009_pid \
    --model akt_pid \
    --batch_size 32 \
    --lr 2e-5 \
    --d_model 512 \
    --n_block 2 \
    --dropout 0.1 \
    --max_iter 500

10.3 Training Process

The training process loads data, initializes the model, trains for up to max_iter epochs with validation after each epoch, saves the best model based on validation AUC, implements early stopping (40 epochs patience), and finally evaluates on test data. For detailed information about the training loop, see Section 12.1.

10.4 Output Files

After training, the following files are created:

Model Files (in model/{model_name}/{dataset_name}/):

• {file_name}_{epoch}: Saved model checkpoint at best epoch

Result Files (in result/{model_name}/{dataset_name}/):

• {file_name}: Text file containing:
  • valid_auc: Validation AUC per epoch
  • train_auc: Training AUC per epoch
  • valid_loss: Validation loss per epoch
  • train_loss: Training loss per epoch
  • valid_accuracy: Validation accuracy per epoch
  • train_accuracy: Training accuracy per epoch

11. Model Variants

11.1 AKT-PID (Rasch Model)

Usage: --model akt_pid

Features:

• Includes Problem ID embeddings (difficult_param)
• Learns difficulty parameters for each problem
• Applies L2 regularization on difficulty parameters
• Modifies question and question-answer embeddings based on difficulty

Architecture Details:

• n_pid > 0: Number of unique problem IDs
• Difficulty parameter: nn.Embedding(n_pid + 1, 1)
• Question embedding with difficulty: q_embed + pid_embed * q_embed_diff
• QA embedding with difficulty: qa_embed + pid_embed * qa_embed_diff
• Regularization loss: (pid_embed ** 2.0).sum() * l2

Suitable For:

• Datasets with problem-level annotations
• When question difficulty modeling is important
• ASSISTments2009 and ASSISTments2017 datasets

11.2 AKT-CID (Non-Rasch Model)

Usage: --model akt_cid

Features:

• No Problem ID embeddings
• Simpler architecture
• Uses only question and answer information

Architecture Details:

• n_pid = -1: No problem ID processing
• Standard question and QA embeddings only
• No difficulty parameter regularization

Suitable For:

• Datasets without problem-level metadata
• When computational efficiency is prioritized
• ASSISTments2015 and Statics datasets

12. Training Details

12.1 Training Loop

1. Data Preparation:

   • Loads CSV files for train/validation/test splits
   • Sequences longer than seqlen are split into multiple sequences
   • Data is padded with zeros to seqlen length

2. Model Initialization:

   • Creates AKT model based on configuration
   • Initializes optimizer (Adam) with specified learning rate
   • Moves model to appropriate device (CPU/GPU)

3. Epoch Training:

   • Shuffles training data
   • Processes data in batches
   • Computes loss (BCE with L2 regularization for PID models)
   • Updates model parameters
   • Computes training metrics (AUC, accuracy, loss)

4. Validation:

   • Evaluates on validation set without gradient computation
   • Computes validation metrics
   • Saves model if validation AUC improves

5. Early Stopping:

   • Monitors validation AUC
   • Stops training if no improvement for 40 epochs
   • Restores best model for testing

6. Testing:

   • Loads best model checkpoint
   • Evaluates on test set
   • Reports final test metrics

12.2 Loss Function

The model uses Binary Cross-Entropy Loss with optional L2 regularization:

• Base Loss: BCEWithLogitsLoss on masked predictions
• Regularization (PID models only): L2 * sum(pid_embedding^2)
• Total Loss: base_loss + regularization_loss

12.3 Evaluation Metrics

• AUC (Area Under ROC Curve): Primary metric for model selection
• Accuracy: Percentage of correct predictions (threshold: 0.5)
• Loss: Binary cross-entropy loss

13. Technical Implementation Details

13.1 Attention Mechanism

The multi-head attention includes:

• Positional Effects: Models temporal decay using distance-based weighting
• Masking: Prevents peeking at future interactions
• Gamma Parameters: Learnable position effect scaling per attention head
• Zero Padding: Applied for certain mask configurations

13.2 Data Processing

• Sequence Splitting: Long sequences are split into chunks of seqlen
• Padding: Sequences shorter than seqlen are zero-padded
• Transposition: Data is transposed from (n_samples, seqlen) to (seqlen, n_samples) for batch processing
• Shuffling: Training data is shuffled each epoch

13.3 Model Saving

• Models are saved only when validation AUC improves
• Previous best models are automatically deleted
• Checkpoints include:
  • Model state dictionary
  • Optimizer state dictionary
  • Epoch number
  • Training loss

14. Troubleshooting

14.1 Common Issues

CUDA Out of Memory:

• Reduce --batch_size
• Reduce --d_model or --d_ff
• Reduce --seqlen (if dataset allows)

Model Not Converging:

• Adjust learning rate (--lr)
• Increase --max_iter
• Check data format matches expected structure

File Not Found Errors:

• Ensure data files are in correct directory structure
• Verify dataset name matches directory name
• Check file naming convention: {dataset}_train{fold}.csv

14.2 Reproducibility

To ensure reproducible results:

• Set --seed to a fixed value
• Use deterministic CUDA operations (enabled by default)
• Ensure data loading order is consistent