Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 104 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,109 @@
# AI Art
AI Art is a plugin for DSS that allows you to generate images from text prompts.
AI Art is a plugin for Dataiku that allows you to generate images from text prompts.

## Usage
For usage instructions, see the [documentation][plugin_documentation].

[plugin_documentation]: https://www.dataiku.com/product/plugins/ai-art/
With this plugin, you will be able to:

- Generate images from a prompt using **Stable Diffusion**
- Generate images from another image and a prompt using **Stable Diffusion**

## How to set up

### CUDA

A CUDA-capable GPU with at least 6 GB of VRAM is recommended.
You can run the plugin without a GPU, but it will be significantly slower.

If you’re using a CUDA GPU, your GPU must support CUDA 11.6, and the NVIDIA drivers must be installed on the server.
For installation instructions, see [NVIDIA Driver Installation Quickstart Guide](https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html).

### Download weights

Before you can use the plugin, you need to download pre-trained weights to a local managed folder. The plugin is designed to work with version 2 of the Stable Diffusion weights, although it may work with other versions and derivatives as well.

For example, if you run the below code in a notebook, it will download the Stability AI version 2.1 weights, and upload them to a new managed folder.

The code requires a code environment with the *huggingface-hub* package installed.

##### Limitations and warnings

- The weights available from Stability AI are licensed under the
CreativeML OpenRAIL++-M license, which restricts usage. You can view the
license [here](https://huggingface.co/stabilityai/stable-diffusion-2/raw/main/LICENSE-MODEL). Hence, the weights must be acquired manually before you can use this plugin.

- The weights must be stored on the local filesystem. If a remote folder (S3,
etc) is used, or if the recipe uses containerized execution, the weights
will be downloaded to a temporary directory every time the recipe is run.
This is because the method used to load the weights ([from_pretrained])
requires a local filepath.

[from_pretrained]: https://huggingface.co/docs/diffusers/v0.6.0/en/api/diffusion_pipeline#diffusers.DiffusionPipeline.from_pretrained



```
import tempfile
import dataiku
import huggingface_hub

# Name of the local folder that will be created
FOLDER_NAME = "my_weights"
# Hugging Face repository
REPO = "stabilityai/stable-diffusion-2-1"
# Revision of the repository
#
# If using the Stability AI weights, the "fp16" revision is recommended
# if using CUDA. Otherwise, use the "main" revision.
REVISION = "fp16"

client = dataiku.api_client()
project = client.get_default_project()

# Create the managed folder
folder = project.create_managed_folder(FOLDER_NAME)

with tempfile.TemporaryDirectory(dir=".") as temp_dir:
# Download the weights to a temp local dir
huggingface_hub.snapshot_download(
repo_id=REPO,
revision=REVISION,
local_dir=temp_dir,
local_dir_use_symlinks=False,
# Skip large files that the plugin doesn't need
ignore_patterns=["*.ckpt", "*.safetensors"],
)

# Upload the weights to the managed folder
folder.upload_folder("/", temp_dir)
```


Using a folder that’s stored on the local filesystem is recommended. If the folder is stored on a remote connection (Amazon S3, Google Cloud Storage, etc), the weights will be downloaded to a temporary directory every time the recipe is run.

## How to use

AI Art contains two methods for generating images: Text-to-Image
Generation, and Text-Guided Image-to-Image Generation

### Text-to-Image Generation

Text-to-Image Generation is used to generate images from a text prompt.

1. Create a *Text-to-Image Generation* recipe with your weights folderas the input.
2. Enter a text prompt in the *Prompt* field.
3. If you’re using the *fp16* revision of the weights, be sure to check the *Half precision* field.

### Text-Guided Image-to-Image Generation

Text-Guided Image-to-Image Generation is used to modify an existing
reference image based on a text prompt.

1. Obtain a reference image that you want to use as a base, and upload it to a managed folder.
2. Create a *Text-Guided Image-to-Image Generation* recipe with your weights folder and your base-image folder as the inputs.
3. Enter a text prompt in the *Prompt* field.
4. Enter the path to your base image in the *Base image* field.
5. If you’re using the *fp16* revision of the weights, be sure to check the *Half precision* field.

## Testing and linting
Format the code using Black:
Expand Down Expand Up @@ -33,7 +132,7 @@ make tests
```

## Building
Create a plugin archive that can be imported into DSS:
Create a plugin archive that can be imported into Dataiku:
```bash
make plugin
```
Expand All @@ -49,4 +148,4 @@ make plugin
This is because the method used to load the weights ([from_pretrained])
requires a local filepath.

[from_pretrained]: https://huggingface.co/docs/diffusers/v0.6.0/en/api/diffusion_pipeline#diffusers.DiffusionPipeline.from_pretrained
[from_pretrained]: https://huggingface.co/docs/diffusers/v0.6.0/en/api/diffusion_pipeline#diffusers.DiffusionPipeline.from_pretrained
Loading