CLEVR Dataset Generation New Version

This is the code used to generate the CLEVR dataset as described in the paper:

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Fei-Fei Li, Larry Zitnick, Ross Girshick
Presented at CVPR 2017

Code and pretrained models for the baselines used in the paper can be found here.

This repository is based on the original CLEVR dataset generation project and has been updated so that image generation can run with Blender 3.6 and CUDA 12.

Main upgrades in this version:

• Image generation supports setting a random seed and controlling the number of generated images.
• Question generation supports setting a random seed, generating only one or more selected question types and adding post prompts.
• Question generation provides an interface for writing Chain-of-Thought (CoT) content for each question type.

You can use this code to render synthetic images and compositional questions for those images, like this:

Q: How many small spheres are there?
A: 2

Q: What number of cubes are small things or red metal objects?
A: 2

Q: Does the metal sphere have the same color as the metal cylinder?
A: Yes

Q: Are there more small cylinders than metal things?
A: No

Q: There is a cylinder that is on the right side of the large yellow object behind the blue ball; is there a shiny cube in front of it?
A: Yes

If you find this code useful in your research then please cite

@inproceedings{johnson2017clevr,
  title={CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning},
  author={Johnson, Justin and Hariharan, Bharath and van der Maaten, Laurens
          and Fei-Fei, Li and Zitnick, C Lawrence and Girshick, Ross},
  booktitle={CVPR},
  year={2017}
}

The original code was developed and tested on OSX and Ubuntu 16.04. This version has been adapted for Blender 3.6, whose bundled Python is 3.10, and can use CUDA 12 compatible NVIDIA drivers for GPU rendering.

Step 1: Generating Images

First we render synthetic images using Blender, outputting both rendered images as well as a JSON file containing ground-truth scene information for each image.

Installing Blender 3.6

Install Blender 3.6 LTS, then add its directory to your shell PATH so later commands can call blender directly. On Linux, download and extract the official Blender 3.6.23 tarball:

wget https://download.blender.org/release/Blender3.6/blender-3.6.23-linux-x64.tar.xz
tar -xf blender-3.6.23-linux-x64.tar.xz

Add the extracted Blender directory to your shell startup file:

echo "export PATH=$(pwd)/blender-3.6.23-linux-x64:\$PATH" >> ~/.bashrc
source ~/.bashrc

If you installed Blender somewhere else, replace $(pwd)/blender-3.6.23-linux-x64 with that path when updating PATH.

Check that Blender is available:

blender --version

Blender ships with its own installation of Python which is used to execute scripts that interact with Blender. Blender 3.6 uses Python 3.10. In most cases render_images.py can now import utils.py directly when run from image_generation; if Blender cannot import it, add the image_generation directory to Blender's bundled Python path with a .pth file:

echo $PWD/image_generation >> blender-3.6.23-linux-x64/3.6/python/lib/python3.10/site-packages/clevr.pth

If Blender was extracted somewhere else, use that Blender directory in the .pth path. For example:

echo $PWD/image_generation >> /path/to/blender-3.6.23-linux-x64/3.6/python/lib/python3.10/site-packages/clevr.pth

Rendering Images

You can then render some images like this:

cd image_generation
blender --background --python render_images.py -- --num_images 10

To make generation reproducible and force an exact object count per image:

blender --background --python render_images.py -- --num_images 10 --random_seed 123 --num_objects 5

On OSX the blender binary is located inside the blender.app directory; for convenience you may want to add the following alias to your ~/.bash_profile file:

alias blender='/Applications/blender/blender.app/Contents/MacOS/blender'

If you have an NVIDIA GPU, CUDA 12, and a driver supported by Blender 3.6, then you can use the GPU to accelerate rendering like this:

blender --background --python render_images.py -- --num_images 10 --use_gpu 1 --gpu_backend CUDA

On RTX cards, --gpu_backend OPTIX is also supported by Blender 3.6 and may be faster.

After a rendering command terminates you should have ten freshly rendered images stored in output/images like these:

The file output/CLEVR_scenes.json will contain ground-truth scene information for all newly rendered images.

You can find more details about image rendering here.

Step 2: Generating Questions

Next we generate questions, functional programs, and answers for the rendered images generated in the previous step. This step takes as input the single JSON file containing all ground-truth scene information, and outputs a JSON file containing questions, answers, and functional programs for the questions in a single JSON file.

You can generate questions like this:

cd question_generation
python generate_questions.py

The file output/CLEVR_questions.json will then contain questions for the generated images.

You can find more details about question generation here.