Evolution Strategies for Deep Reinforcement Learning Pretraining

A comprehensive research project investigating Evolution Strategies as pretraining mechanisms for Deep Reinforcement Learning algorithms

École Polytechnique Fédérale de Lausanne (EPFL)
In collaboration with Swiss Data Science Center

📊 Research Poster

Click the badge above to view the full research poster (PDF)

📄 Full Paper: RL_project.pdf

Demo Video

MuJoCo Humanoid Agent Trained with Evolution Strategies

https://github.com/MarioRicoIbanez/ES_DRL/assets/sim.mp4

🎥 Simulation Video: sim.mp4 - Watch the trained agent in action!

📝 Abstract

Although Deep Reinforcement Learning has proven highly effective for complex decision-making problems, it demands significant computational resources and careful parameter adjustment in order to develop successful strategies. Evolution strategies offer a more straightforward, derivative-free approach that is less computationally costly and simpler to deploy. However, ES generally do not match the performance levels achieved by DRL, which calls into question their suitability for more demanding scenarios.

This study examines the performance of ES and DRL across tasks of varying difficulty, including Flappy Bird, Breakout, and MuJoCo environments, as well as whether ES could be used for initial training to enhance DRL algorithms.

Key Findings:

✅ ES accelerates early learning in simple environments (Flappy Bird)
❌ ES struggles with high-dimensional state spaces (Breakout)
⚠️ ES pretraining shows limited benefit for complex tasks (MuJoCo)
📈 DRL algorithms achieve better final performance but require more tuning

🎯 At a Glance

🔬 Research Focus

Comparing ES vs DRL performance
Evaluating ES as pretraining strategy
Testing across 3 environment types
Analyzing training speed & robustness

🛠️ Technologies Used

Physics: Brax, MuJoCo
Frameworks: JAX, NumPy
Algorithms: ES, DQN, PPO, TD3
Tracking: Weights & Biases

📊 Environments Tested

Discrete: Flappy Bird, Breakout
Continuous: Hopper, Walker2d, HalfCheetah
Dimensions: 2D to 84×84×4
Complexity: Simple to High

✨ Key Contributions

Comprehensive ES vs DRL comparison
ES pretraining analysis
Architecture compatibility insights
Reproducible benchmark suite

🎯 Research Motivation

Traditional Deep Reinforcement Learning approaches often require:

Extensive training time and computational resources
Careful hyperparameter tuning for convergence
Gradient computation through complex backpropagation

Evolution Strategies offer an alternative:

✨ Gradient-free optimization
🚀 Simpler implementation and parallelization
💰 Lower computational cost per update
🎲 Robust exploration through parameter space perturbations

Research Questions:

Can ES reach intermediate performance benchmarks faster than DRL?
Can ES serve as effective pretraining to improve DRL training speed and robustness?

📄 Research Materials

This repository contains the complete implementation of the research presented in our paper:

Resource	Description	Link
📘 Full Paper	Complete research paper with detailed methodology and analysis	RL_project.pdf
🎨 Research Poster	Visual summary of findings presented at EPFL	Poster PDF
🎥 Demo Video	MuJoCo Humanoid agent trained with ES	sim.mp4
💻 Source Code	Full implementation with experiments	This repository

Publication Details:

Institution: École Polytechnique Fédérale de Lausanne (EPFL)
Collaboration: Swiss Data Science Center
Year: 2025
Conference: NeurIPS 2024 Format

🧮 Mathematical Formulation

Evolution Strategies

Evolution Strategies optimize a smoothed version of the objective function through parameter perturbations:

Objective Function:

$$\tilde{J}(\theta) = \mathbb{E}_{\epsilon \sim \mathcal{N}(0, I)}[F(\theta + \sigma \epsilon)]$$

where:

$\theta$ = policy parameters
$F(\theta)$ = total reward (fitness) for parameters $\theta$
$\sigma$ = noise standard deviation
$\epsilon$ = Gaussian perturbation vector

Gradient Estimation:

Using the log-likelihood trick, the gradient can be estimated as:

$$\nabla_\theta \tilde{J}(\theta) = \frac{1}{\sigma} \mathbb{E}_{\epsilon} \left[ F(\theta + \sigma \epsilon) \cdot \epsilon \right]$$

Monte Carlo Approximation:

With $n$ independent samples $\epsilon_1, \dots, \epsilon_n$:

$$\nabla_\theta \tilde{J}(\theta) \approx \frac{1}{n \sigma} \sum_{i=1}^n F(\theta + \sigma \epsilon_i) \cdot \epsilon_i$$

Update Rule:

$$\theta_{t+1} = \theta_t + \alpha \cdot \nabla_\theta \tilde{J}(\theta_t)$$

where $\alpha$ is the learning rate.

Algorithm Pseudocode:

# Evolution Strategies Training Loop
def train_ES(policy, population_size, sigma, learning_rate, num_generations):
    theta = initialize_parameters(policy)
    
    for generation in range(num_generations):
        # 1. Generate population
        perturbations = [sample_gaussian() for _ in range(population_size)]
        
        # 2. Evaluate fitness
        rewards = []
        for epsilon in perturbations:
            theta_perturbed = theta + sigma * epsilon
            reward = evaluate_policy(theta_perturbed)
            rewards.append(reward)
        
        # 3. Estimate gradient
        gradient = (1 / (population_size * sigma)) * sum(
            reward * epsilon for reward, epsilon in zip(rewards, perturbations)
        )
        
        # 4. Update parameters
        theta = theta + learning_rate * gradient
    
    return theta

Key Advantages:

🚀 No backpropagation required
🔄 Highly parallelizable
📊 Variance independent of episode length
🎯 Effective in sparse reward settings

Deep Q-Networks (DQN)

DQN approximates the optimal action-value function using a neural network:

Bellman Optimality Equation:

$$Q^*(s, a) = \mathbb{E}_{s'} \left[ r + \gamma \max_{a'} Q^*(s', a') \,|\, s, a \right]$$

Loss Function:

$$L(\theta) = \mathbb{E}_{(s,a,r,s') \sim D} \left[ \left( y - Q(s, a; \theta) \right)^2 \right]$$

where the target is:

$$y = r + \gamma \max_{a'} Q(s', a'; \theta^{-})$$

and $\theta^{-}$ are parameters of the target network.

Proximal Policy Optimization (PPO)

PPO uses a clipped surrogate objective for stable policy updates:

Clipped Objective:

$$L^{CLIP}(\theta) = \hat{\mathbb{E}}_t[\min(r_t(\theta)\hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t)]$$

where:

$r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ is the probability ratio
$\hat{A}_t$ is the advantage estimate
$\epsilon$ is the clipping parameter (typically 0.2)

🏗️ Architecture & Implementation

Project Structure

ES_DRL/
├── src/es_drl/
│   ├── main_es.py              # Main training script for ES
│   ├── main_finetune.py        # Main script for DRL fine-tuning
│   ├── es/
│   │   ├── base.py             # Base ES class with common functionality
│   │   ├── basic_es.py         # Basic ES implementation
│   │   ├── brax_training_utils.py  # Training utilities for Brax
│   │   ├── ppo.py              # PPO implementation
│   │   ├── ppo_training_utils.py   # PPO training utilities
│   │   ├── pretraining.py      # ES + DRL pretraining strategy
│   │   └── tr_es.py            # Trust-region ES variant
│   ├── dpg/
│   │   └── td3_trainer.py      # TD3 implementation
│   └── utils/
│       ├── callbacks.py        # Training callbacks
│       └── logger.py           # Logging utilities
├── configs/
│   ├── common.yaml             # Common configuration
│   ├── es/                     # ES-specific configs
│   │   ├── basic_es.yaml
│   │   ├── ppo.yaml
│   │   ├── pretraining.yaml
│   │   └── tr_es.yaml
│   ├── td3/                    # TD3-specific configs
│   │   └── td3_finetune.yaml
│   └── [environment].yaml      # Environment-specific configs
├── scripts/
│   ├── env.sh                  # Environment setup
│   ├── run_es.sh               # ES training script
│   └── run_finetune.sh         # Fine-tuning script
├── results/                    # Training results and plots
├── requirements.txt            # Python dependencies
├── RL_project.pdf             # Full research paper
└── Evolution_Strategies_for_Deep_RL_pretraining___Poster.pdf

Neural Network Architectures

Flappy Bird & Breakout (Discrete Actions)

Policy Network: Fully connected MLP
- Hidden layers: [64, 64] with Tanh activation
- Output: Q-values for each discrete action
- Parameters: ~4,000 (Flappy Bird), ~850,000 (Breakout CNN)

MuJoCo Environments (Continuous Control)

Actor Network: Fully connected MLP
- Hidden layers: [32, 32, 32, 32]
- Output: Mean and std for action distribution
- Parameters: ~10,000 - 15,000 depending on environment
Critic Network (PPO only):
- Same architecture as actor
- Output: Single value estimate

🚀 Getting Started

Prerequisites

Python 3.8+
CUDA-capable GPU (recommended for faster training)
8GB+ RAM

Installation

Clone the repository:

git clone https://github.com/MarioRicoIbanez/ES_DRL.git
cd ES_DRL

Create a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install dependencies:

pip install -r requirements.txt

Verify installation:

python -c "import jax; import brax; print('✓ Installation successful!')"

⚡ Quick Start Commands

Train an agent in 30 seconds:

# Evolution Strategies on Walker2d
python src/es_drl/main_es.py --config configs/es/basic_es.yaml --env_config configs/walker2d.yaml --seed 42

# PPO on Hopper
python src/es_drl/main_es.py --config configs/es/ppo.yaml --env_config configs/hopper.yaml --seed 42

# ES Pretraining → PPO Fine-tuning
python src/es_drl/main_es.py --config configs/es/pretraining.yaml --env_config configs/hopper.yaml --seed 42

📋 Detailed Usage

Training with Evolution Strategies

# Basic ES training on Walker2d
python src/es_drl/main_es.py \
    --config configs/es/basic_es.yaml \
    --env_config configs/walker2d.yaml \
    --seed 42

# ES training on Hopper
python src/es_drl/main_es.py \
    --config configs/es/basic_es.yaml \
    --env_config configs/hopper.yaml \
    --seed 42

Training with PPO (DRL)

# PPO training on Walker2d
python src/es_drl/main_es.py \
    --config configs/es/ppo.yaml \
    --env_config configs/walker2d.yaml \
    --seed 42

Pretraining with ES + Fine-tuning with PPO

# Step 1: Pretrain with ES
python src/es_drl/main_es.py \
    --config configs/es/pretraining.yaml \
    --env_config configs/hopper.yaml \
    --seed 42

# Step 2: Fine-tune with PPO (automatically loads ES weights)
python src/es_drl/main_finetune.py \
    --config configs/td3/td3_finetune.yaml \
    --env_config configs/hopper.yaml \
    --pretrained_path results/es/hopper/checkpoint.pkl \
    --seed 42

Using Shell Scripts

# Set up environment variables
source scripts/env.sh

# Run ES training
bash scripts/run_es.sh walker2d 42

# Run fine-tuning
bash scripts/run_finetune.sh walker2d 42

⚙️ Configuration

Configuration Files

Training is configured through YAML files with hierarchical structure:

Common Settings (configs/common.yaml):

training:
  num_timesteps: 1000000
  episode_length: 1000
  num_envs: 2048
  
logging:
  wandb_project: "es-drl"
  log_interval: 100
  save_interval: 1000

ES-Specific (configs/es/basic_es.yaml):

algorithm:
  name: "basic_es"
  sigma: 0.05              # Noise standard deviation
  population_size: 128     # Number of perturbations per iteration
  learning_rate: 0.01      # Update step size
  elite_fraction: 0.2      # Top performers for fitness shaping
  
network:
  hidden_sizes: [32, 32, 32, 32]
  activation: "tanh"

PPO-Specific (configs/es/ppo.yaml):

algorithm:
  name: "ppo"
  learning_rate: 3e-4
  batch_size: 512
  num_minibatches: 32
  num_updates_per_batch: 64
  gamma: 0.997             # Discount factor
  gae_lambda: 0.95         # GAE parameter
  clip_epsilon: 0.2        # PPO clipping parameter
  entropy_coef: 0.0
  value_coef: 0.5

Environment-Specific (configs/walker2d.yaml):

environment:
  name: "walker2d"
  backend: "brax"          # Physics engine
  reward_scaling: 5.0
  
  # Environment parameters
  ctrl_cost_weight: 1e-3
  forward_reward_weight: 1.0
  healthy_reward: 1.0
  healthy_z_range: [0.8, 2.0]

Key Hyperparameters

Parameter	ES	PPO	Description
`population_size`	128	-	Number of parameter perturbations
`sigma`	0.05	-	Noise standard deviation for ES
`learning_rate`	0.01	3e-4	Step size for updates
`num_envs`	-	2048-8192	Parallel environments for PPO
`gamma`	0.99	0.997	Discount factor
`clip_epsilon`	-	0.2	PPO clipping parameter

🧪 Experimental Results

Environments Tested

Environment	Type	State Dim	Action Dim	Complexity
Flappy Bird	Discrete	Low (~10)	2	Simple
Breakout	Discrete	High (84×84×4)	4	Medium
Hopper	Continuous	11	3	Medium
Walker2d	Continuous	17	6	High
HalfCheetah	Continuous	17	6	High

Key Findings

1. Flappy Bird (Simple Environment)

✅ ES shows strong early performance

Quickly finds stable policies
Reaches intermediate rewards ~5x faster than DQN
Effective pretraining for DQN acceleration

📊 Results:

ES: Converges in ~100 episodes
DQN: Requires ~500 episodes for similar performance
ES→DQN: Reaches high rewards ~2x faster than DQN alone

2. Breakout (High-Dimensional State Space)

❌ ES struggles with complexity

Plateaus at mean reward ~1.5 (image-based)
Limited improvement even with large populations (n=50)
RAM-based input (128-D) only marginally better (reward ~4)

🎯 DQN outperforms:

Achieves mean reward ~30
CNN architecture effectively processes visual input
Higher variance but better final performance

3. MuJoCo Environments (Continuous Control)

⚠️ Mixed results for ES pretraining

HalfCheetah:

✅ PPO converges 20× faster than ES
❌ ES pretraining provides no benefit

Walker2d:

❌ PPO fails to converge consistently
⚠️ ES shows stable but slow learning
❌ ES pretraining doesn't improve PPO robustness

Hopper:

⚠️ PPO shows high variance across seeds
✅ ES demonstrates more stable training
❌ ES pretraining doesn't enhance PPO performance

Performance Comparison Summary

Algorithm Performance Across Key Metrics

Method	Training Speed	Final Performance	Stability	Hyperparameter Sensitivity	Best Use Case
ES	🟡 Moderate	🟡 Moderate	🟢 High	🟢 Low	Simple environments, initial exploration
DQN/PPO	🟢 Fast*	🟢 High	🟡 Moderate	🔴 High	Complex tasks when tuned
ES→DRL	🟢 Fast**	🟢 High	🟡 Moderate	🟡 Moderate	Simple to medium complexity

^{*When properly tuned | **Only in simple environments}

Detailed Performance by Environment

Environment	ES Performance	DRL Performance	ES Pretraining Benefit	Winner
Flappy Bird	🟢 Good	🟢 Excellent	✅ Significant	ES→DQN
Breakout	🔴 Poor	🟢 Excellent	❌ None	DQN
HalfCheetah	🟡 Moderate	🟢 Excellent	❌ None	PPO
Hopper	🟡 Moderate	🟡 Unstable	❌ None	ES (stability)
Walker2d	🟡 Moderate	🔴 Poor	❌ None	ES

Legend:

🟢 Excellent/High - 🟡 Moderate/Medium - 🔴 Poor/Low
✅ Beneficial - ❌ Not beneficial - ⚠️ Conditional

📈 Monitoring & Logging

Weights & Biases Integration

All experiments are tracked using W&B:

# Automatic logging includes:
- Episode rewards (mean, std, min, max)
- Training time and timesteps
- Parameter statistics (norm, updates)
- Environment-specific metrics
- Videos of agent behavior

View logs:

wandb login
# Logs automatically sync to your W&B project

Local Logging

CSV files are saved to results/[algorithm]/[environment]/:

results/
└── es/
    └── walker2d/
        ├── basic_es_seed42.csv
        ├── basic_es_seed42.png
        └── checkpoints/
            └── generation_1000.pkl

Visualization

Generate training plots:

python scripts/plot_results.py \
    --results_dir results/es/walker2d \
    --output plots/walker2d_comparison.png

🔬 Reproducing Paper Results

Flappy Bird Experiments

# ES training
python src/es_drl/main_es.py \
    --config configs/es/basic_es.yaml \
    --env_config configs/flappy_bird.yaml \
    --seed 42

# DQN training  
python src/es_drl/main_es.py \
    --config configs/dqn/dqn.yaml \
    --env_config configs/flappy_bird.yaml \
    --seed 42

# ES pretraining + DQN fine-tuning
python src/es_drl/main_es.py \
    --config configs/es/pretraining.yaml \
    --env_config configs/flappy_bird.yaml \
    --seed 42

Breakout Experiments

# ES on RAM-based input
python src/es_drl/main_es.py \
    --config configs/es/basic_es.yaml \
    --env_config configs/breakout_ram.yaml \
    --seed 42

# DQN on image-based input
python src/es_drl/main_es.py \
    --config configs/dqn/dqn_cnn.yaml \
    --env_config configs/breakout.yaml \
    --seed 42

MuJoCo Experiments

# Run all MuJoCo experiments with different seeds
for env in hopper walker2d halfcheetah; do
    for seed in 42 123 456; do
        # ES
        python src/es_drl/main_es.py \
            --config configs/es/basic_es.yaml \
            --env_config configs/${env}.yaml \
            --seed ${seed}
        
        # PPO
        python src/es_drl/main_es.py \
            --config configs/es/ppo.yaml \
            --env_config configs/${env}.yaml \
            --seed ${seed}
        
        # ES→PPO pretraining
        python src/es_drl/main_es.py \
            --config configs/es/pretraining.yaml \
            --env_config configs/${env}.yaml \
            --seed ${seed}
    done
done

💡 Key Insights & Takeaways

When to Use Evolution Strategies

✅ ES is effective for:

Simple environments with low-dimensional state spaces
Sparse or deceptive reward landscapes
Quick prototyping and initial exploration
Parallel computation with limited communication
Non-differentiable environments

❌ ES struggles with:

High-dimensional observations (images, complex sensors)
Environments requiring fine-grained control
Tasks demanding very high final performance
Long-horizon temporal dependencies

Pretraining Recommendations

Effective pretraining requires:

✅ Environment simplicity (low-dimensional states)
✅ Architectural compatibility (ES and DRL networks must align)
✅ Sparse rewards where ES excels at initial exploration
❌ Not recommended for actor-critic methods (PPO, SAC) where ES only optimizes actor

Architectural Considerations

Limitation: ES pretraining is less effective for actor-critic algorithms (PPO, SAC) because:

ES optimizes a single policy network
Actor-critic methods require separate value functions
Transferring only the actor may not provide sufficient benefit

Future directions:

Co-evolving actor and critic networks
Architecture-aware hybrid approaches
Adaptive transfer strategies

👥 Authors

École Polytechnique Fédérale de Lausanne (EPFL)

Adrian Martínez López _EPFL	Ananya Gupta _EPFL	Hanka Goralija _EPFL
Mario Rico Ibáñez _EPFL	Saúl Fenollosa Arguedas _EPFL	Tamar Alphaidze _EPFL

🤝 Collaboration

This research was conducted in collaboration with the Swiss Data Science Center (SDSC), a joint venture between EPFL and ETH Zurich. The SDSC provided:

🎓 Academic Supervision: Expert guidance on research methodology
💻 Computational Resources: Access to high-performance computing infrastructure
📊 Data Science Expertise: Technical support and best practices
🌐 Research Network: Connections to the broader data science community

About Swiss Data Science Center:

The Swiss Data Science Center develops and deploys cutting-edge data science and machine learning techniques to help solve complex societal challenges. By bridging the gap between academic research and real-world applications, SDSC enables researchers to leverage state-of-the-art tools and methodologies.

🔗 Learn more: www.datascience.ch

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

We would like to express our gratitude to:

🏛️ Swiss Data Science Center (SDSC) for their invaluable collaboration, computational resources, and technical guidance throughout this research project
🎓 École Polytechnique Fédérale de Lausanne (EPFL) for providing the academic environment and infrastructure
🔬 Brax Team at Google Research for the differentiable physics engine that enabled efficient experimentation
🎮 OpenAI Gym for standardized RL environment interfaces
📚 Stable-Baselines3 team for reference implementations and best practices
🌐 JAX Team for the high-performance numerical computing framework
📊 Weights & Biases for experiment tracking and visualization tools
💡 The broader reinforcement learning research community for open-source tools and reproducible research practices

Special Thanks:

To our reviewers and colleagues who provided valuable feedback
To the open-source community for making this research possible
To all contributors and users of this codebase

📖 References

Salimans, T., et al. (2017). "Evolution Strategies as a Scalable Alternative to Reinforcement Learning." arXiv preprint arXiv:1703.03864.
Mnih, V., et al. (2013). "Playing Atari with Deep Reinforcement Learning." arXiv preprint arXiv:1312.5602.
Schulman, J., et al. (2017). "Proximal Policy Optimization Algorithms." arXiv preprint arXiv:1707.06347.
Todorov, E., et al. (2012). "MuJoCo: A physics engine for model-based control." IROS 2012.
Freeman, C. D., et al. (2021). "Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation." NeurIPS 2021 Datasets and Benchmarks Track.

🐛 Troubleshooting

Common Issues

1. CUDA out of memory

# Reduce number of parallel environments
# In config file: num_envs: 1024  # instead of 8192

2. Wandb authentication

wandb login
# Enter your API key from https://wandb.ai/authorize

3. MuJoCo installation issues

# Install MuJoCo dependencies
sudo apt-get install libglew-dev patchelf

4. Slow training

# Enable JAX GPU acceleration
pip install --upgrade "jax[cuda]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

❓ Frequently Asked Questions

Q: When should I use ES instead of DRL?

Use ES when:

🎯 You have a simple environment with low-dimensional states
🔍 You need quick initial exploration
💻 You have access to many parallel workers
🎲 Your reward function is sparse or deceptive

Use DRL when:

🖼️ You have high-dimensional observations (images)
🎯 You need the highest possible performance
⏱️ Sample efficiency is critical
🔧 You can afford hyperparameter tuning

Q: Why doesn't ES pretraining help PPO in MuJoCo?

ES only optimizes a single policy network, while PPO uses separate actor and critic networks. When we pretrain with ES:

✅ We get a reasonable actor initialization
❌ The critic still starts from scratch
⚠️ The architectural mismatch limits knowledge transfer

Future work: Co-evolution of actor and critic networks may address this limitation.

Q: How do I choose hyperparameters?

For ES:

Start with sigma=0.05, population_size=128, lr=0.01
Increase population size for complex environments
Reduce sigma if training is unstable

For PPO:

Use configs from configs/es/ppo.yaml as baseline
Adjust num_envs based on available memory
Tune learning rate if training diverges

General advice: ES is more robust to hyperparameters than PPO!

Q: Can I use this for my own environments?

Yes! The code is designed to be extensible:

Create a new environment config in configs/your_env.yaml
Ensure your environment follows the Gym API
Run training with your config

python src/es_drl/main_es.py --config configs/es/basic_es.yaml --env_config configs/your_env.yaml

For Brax environments, no changes needed. For custom environments, you may need to add a wrapper.

Q: How can I visualize my trained agents?

During training:

Enable video recording in your config
Videos are saved to results/[algorithm]/[env]/videos/

After training:

python scripts/visualize_agent.py \
    --checkpoint results/es/hopper/checkpoint.pkl \
    --env hopper \
    --num_episodes 5

Q: What computational resources do I need?

Minimum:

8GB RAM
4 CPU cores
Training time: Hours to days

Recommended:

16GB+ RAM
GPU with 8GB+ VRAM
8+ CPU cores for parallelization
Training time: Minutes to hours

Note: ES benefits more from CPU parallelization, while DRL benefits more from GPU acceleration.

Q: Where can I find more details about the experiments?

📄 Full paper: RL_project.pdf - Contains complete methodology, results, and analysis

🎨 Poster: Poster PDF - Visual summary of key findings

📊 Code: Browse src/ for implementation details

📈 Results: Check results/ for training curves and checkpoints

🔄 Future Work

Implement co-evolution of actor and critic networks
Test on more complex environments (Humanoid, manipulation tasks)
Explore trust-region ES variants
Integrate with other DRL algorithms (SAC, TD3)
Develop architecture-aware transfer learning
Multi-task and transfer learning experiments

💬 Getting Help & Contributing

Need Help?

📖 Documentation: Start with this README and the full paper
❓ Issues: Open an issue on GitHub for bugs or questions
💬 Discussions: Use GitHub Discussions for general questions
📧 Contact: Reach out to the authors for research-related inquiries

Contributing

We welcome contributions! Here's how you can help:

🐛 Report bugs via GitHub Issues
💡 Suggest features or improvements
🔧 Submit pull requests with bug fixes or enhancements
📝 Improve documentation or add examples
🌟 Star the repo if you find it useful!

Contribution Guidelines:

Follow existing code style
Add tests for new features
Update documentation as needed
Reference related issues in PRs

Research Collaboration

Interested in collaborating or extending this research? We're open to:

🔬 Joint research projects
🎓 Supervising thesis projects
🤝 Industry partnerships
📊 Applying these methods to new domains

Contact the authors or Swiss Data Science Center for collaboration opportunities.

For questions, collaboration opportunities, or research inquiries, please open an issue or contact the authors.

_{Made with ❤️ at EPFL in collaboration with Swiss Data Science Center}

Name		Name	Last commit message	Last commit date
Latest commit History 62 Commits
configs		configs
results/es/walker2d		results/es/walker2d
scripts		scripts
src/es_drl		src/es_drl
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
Evolution_Strategies_for_Deep_RL_pretraining___Poster.pdf		Evolution_Strategies_for_Deep_RL_pretraining___Poster.pdf
README.md		README.md
RL_project.pdf		RL_project.pdf
mujoco_humanoid.py		mujoco_humanoid.py
optimize_ES.py		optimize_ES.py
requirements.txt		requirements.txt
sim.mp4		sim.mp4

Folders and files

Latest commit

History

Repository files navigation

Evolution Strategies for Deep Reinforcement Learning Pretraining

📑 Table of Contents

📊 Research Poster

Demo Video

📝 Abstract

🎯 At a Glance

🎯 Research Motivation

📄 Research Materials

🧮 Mathematical Formulation

Evolution Strategies

Deep Q-Networks (DQN)

Proximal Policy Optimization (PPO)

🏗️ Architecture & Implementation

🏗️ Architecture & Implementation

Project Structure

Neural Network Architectures

Flappy Bird & Breakout (Discrete Actions)

MuJoCo Environments (Continuous Control)

🚀 Getting Started

Prerequisites

Installation

⚡ Quick Start Commands

📋 Detailed Usage

Training with Evolution Strategies

Training with PPO (DRL)

Pretraining with ES + Fine-tuning with PPO

Using Shell Scripts

⚙️ Configuration

Configuration Files

Key Hyperparameters

🧪 Experimental Results

Environments Tested

Key Findings

1. Flappy Bird (Simple Environment)

2. Breakout (High-Dimensional State Space)

3. MuJoCo Environments (Continuous Control)

Performance Comparison Summary

Detailed Performance by Environment

📈 Monitoring & Logging

Weights & Biases Integration

Local Logging

Visualization

🔬 Reproducing Paper Results

Flappy Bird Experiments

Breakout Experiments

MuJoCo Experiments

💡 Key Insights & Takeaways

When to Use Evolution Strategies

Pretraining Recommendations

Architectural Considerations

👥 Authors

🤝 Collaboration

📄 License

🙏 Acknowledgments

📖 References

🐛 Troubleshooting

Common Issues

❓ Frequently Asked Questions

🔄 Future Work

💬 Getting Help & Contributing

Need Help?

Contributing

Research Collaboration

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages