Reinforcment Learning (RL) in CARLA

Overview

This repository features a custom CARLA Gymnasium environment designed for reinforcement learning research, as part of my undergraduate thesis on reward design in reinforcement learning for autonomous vehicles. This environment extends prior work by Jianyu Chen, upgraded to leverage Gymnasium for improved compatibility and functionality, and incorporates Stable-Baselines3 (SB3) for advanced reinforcement learning capabilities. As well, the code has been heavily modularized and improved with additional features as detailed in the following section.

A special thanks to my thesis supervisor, Professor Leilani Gilpin, and my AIEA Lab peers, Vik Dhillon and Dominick Rangel, for their support and guidance throughout this research.

Features

• Scalable observation space: Supports state vectors and multi-camera sensor inputs.
• Reward function design: An 8-component reward function balancing safety (e.g., lane adherence, collision avoidance) and efficiency (e.g., waypoint completion). Currently only uses state vector observations, but the cameras are readily optimized for future RL applications.
• Visualization: Four camera sensors incorporated for enhanced visualization, rendered through Pygame. Added upon the existing bird's-eye feature from the original gym environment.
• Training & monitoring: RL agents trained using SB3's Deep Q-Networks (DQN) and multi-input policy, with performance tracked using TensorBoard.

Setup

1. Set up the platform and install CARLA.

This project can be run locally on Windows or through a Linux-based GUI on the Nautilus cluster to manage large-scale simulations. Follow this tutorial for cluster setup. Once successful, follow the CARLA installation guide to install CARLA via the GUI's terminal.

2. Upon up a second terminal, clone the repository and download dependencies.

git clone https://github.com/calstephano/CARLA-Gymnasium-RL.git
cd CARLA-Gymnasium-RL
pip3 install -r requirements.txt
pip3 install -e .
export PYTHONPATH=$PYTHONPATH:<path to CARLA>/PythonAPI/carla/dist/carla-<version here>-py3

3. Return to the first terminal and host CARLA.

The first terminal should already be at CARLA's installation directory and if not, navigate there. Host CARLA through the following command:

./CarlaUE4.sh -carla-rpc-port=2000 -norelativemousemode

4. Return to the second terminal and run the project.

python3 run.py

Viewing Data

TensorBoard is a visualization tool used to monitor training metrics, such as rewards and losses. To launch TensorBoard, open a new terminal and run the following command in the main directory:

tensorboard --logdir ./logs --port 6006

Open your browser and go to:

http://localhost:6006