You need to first ensure that all the necessary drivers (CUDA most important) for the Nvidia GPU are installed. This post will NOT cover how to do that.
What i am working with here is a Nvidia GTX 1070, just so you know.
Preparing the project directory and virtual environment
Create a new directory/folder (whichever terminology your familiar with) and cd to it from a terminal instance.
Now pay attention here.
Diffusers uses PyTorch as its backend. PyTorch has several wheel repositories for different CUDA versions, and you need to choose for Pascal GPUs the one compatible with CUDA 12.1 or 12.4.
To get the right working libraries, we need to fetch them from the repository https://download.pytorch.org/whl/cu124 using the --index-url pip CLI parameter/argument.
Now, to the Diffusers Framework.
Once the PyTorch related things are done,
the only thing to watch out for is ensuring that transformers is fixed at version 5.5.0 .
Why is that? My hunch (and days of trying to figure out why my setup broke when i updated my libraries in my virtual environment),
parts of the transformers API was changed in later version, and diffusers didn't take that into account in their newer versions.
From my findings, version 5.5.0 was the last working one. Since we want this to just work,
make sure that you only get THAT version until diffusers decides to fix this (NOTE: i think they did fix it, but i haven't tested it yet on my GTX 1070 setup).
Now, lets get to what to run inside our project directory.
python3 -m venv ./venv/
./venv/bin/pip install torch torchvision --index-url 'https://download.pytorch.org/whl/cu124'
./venv/bin/pip install diffusers transformers==5.5.0 accelerate peft
This should be all that is needed to get a working virtual environment setup.
Using Diffusers for running Stable Diffusion XL models
This is now the part in which we need to interact with the Diffusers API. In this case, a regular Python script will do the job, and so, i build one, and will share it with you.
#!./venv/bin/python3
import argparse
from pathlib import Path
import torch
from diffusers import (
DPMSolverMultistepScheduler,
StableDiffusionXLPipeline,
)
# Constants.
# ==========
IMAGES_BATCH_COUNT = 2
TORCH_DTYPE = torch.bfloat16
# Prepare and parse the terminal arguments.
# =========================================
args_parser = argparse.ArgumentParser(
description="Custom program for text-to-image using Stable Diffusion XL models."
)
args_parser.add_argument(
"--model",
type=str,
required=True,
help="Huggingface repository path to the SDXL model weights.",
)
args_parser.add_argument(
"--prompt-path",
type=Path,
default=Path("./prompt.txt"),
help="Filepath to the prompt on what to generate.",
)
args_parser.add_argument(
"--negative-prompt-path",
type=Path,
default=None,
help="Filepath to the prompt on what NOT to generate. Ignored if not set.",
)
args_parser.add_argument(
"--sampling-steps",
type=int,
default=26,
help="How many steps are performed in the denoising procedure. More usually result in better image quality.",
)
args_parser.add_argument(
"--guidance-scale",
type=float,
default=6.5,
help="How close it should follow the prompt. Lower values result in more freedom, while higher ones result in more consistency.",
)
args_parser.add_argument(
"--images-count",
type=int,
default=IMAGES_BATCH_COUNT,
help="How many images should be generated per prompt.",
)
args_parser.add_argument(
"--seed",
type=int,
default=None,
help="The seed used to set the state of the PRNG. If not set, chooses a random one.",
)
args_parser.add_argument(
"--output-dirpath",
type=Path,
default=Path("./txt2img_batch"),
help="Where to store the generated images.",
)
parsed_args = args_parser.parse_args()
# Prepare the text-to-image pipeline to generate images.
# ======================================================
prompt = parsed_args.prompt_path.read_text()
negative_prompt = None if parsed_args.negative_prompt_path == None else parsed_args.negative_prompt_path.read_text()
pipeline = StableDiffusionXLPipeline.from_single_file(
parsed_args.model,
custom_pipeline="lpw_stable_diffusion_xl",
torch_dtype=TORCH_DTYPE,
variant="bfp16",
safety_checker=None,
)
pipeline.scheduler = DPMSolverMultistepScheduler.from_config(
pipeline.scheduler.config,
use_karras_sigmas=True,
algorithm_type="dpmsolver++",
)
pipeline.enable_model_cpu_offload()
pipeline.enable_attention_slicing(slice_size="auto")
pipeline.vae.enable_slicing()
# Generate and save the images.
# =============================
batch_count, leftover_count = divmod(parsed_args.images_count, IMAGES_BATCH_COUNT)
images_batch_sequence = []
if batch_count > 0:
images_batch_sequence += [IMAGES_BATCH_COUNT] * batch_count
if leftover_count > 0:
images_batch_sequence += [leftover_count]
generator = None if parsed_args.seed == None else torch.manual_seed(parsed_args.seed)
parsed_args.output_dirpath.mkdir(
parents=True,
exist_ok=True,
)
for num_images_per_prompt in images_batch_sequence:
inference_result = pipeline(
prompt=prompt,
negative_prompt=negative_prompt,
num_inference_steps=parsed_args.sampling_steps,
guidance_scale=parsed_args.guidance_scale,
num_images_per_prompt=num_images_per_prompt,
generator=generator,
)
file_counter = 0
saved_image_paths = []
for generated_image in inference_result.images:
output_filepath = parsed_args.output_dirpath.joinpath(f"image{file_counter}.png")
while output_filepath.exists():
file_counter += 1
output_filepath = parsed_args.output_dirpath.joinpath(f"image{file_counter}.png")
generated_image.save(output_filepath)
saved_image_paths.append(output_filepath.name)
print(f"Saved the generated {num_images_per_prompt} image/s {saved_image_paths} into the directory '{parsed_args.output_dirpath}'.")
This is what i use to generate AI images with Stable Diffusion XL models. It has a solid good enough CLI interface, and the script can be adapted for ones individual needs if something is insufficient.
One important thing to note. Most Stable Diffusion XL models are around 7 GB in size. Add the pipeline components, and we are talking 8+ GB of VRAM taken.
So reducing the VRAM usage is a must, and this is usually done with offloading, attention slicing and VAE slicing.
For optimal speed, i recommend using model offloading, but keep in mind that the whole model is loaded, with the pipeline components being swapped out as needed. This means, it will use more VRAM, and if you are short on it, get ready for out of memory exceptions. In the case of my GTX 1070, i need to ensure that almost nothing is filling the GPUs VRAM other then Diffusers doing its thing, otherwise the script will likely exit with a out of memory exception when saving the generated images, to give you an idea.
Now, to save on VRAM as much as possible (and sacrificing some extra speed), use cpu offloading. This will make it run even on 6 GB of VRAM, but as warned, will take longer.
Best i have found (speed and reliability wise) is using group offloading with CUDA streaming enabled, but this required having more RAM. In my experience, 16 GB is not enough for Stable Diffusion XL models. You need around 24 GB minimum.
Regarding attention slicing, due to PyTorch being kinda a PITA in dealing with Pascal based Nvidia GPU's, you must enable it using the enable_attention_slicing method from the base class DiffusionPipeline. Why is that? Due to FlashAttention (the enabled default on newer PyTorch builds) simply not working (atleast on my GPU), xFormers not working properly (atleast on my GPU, again), so...was left with only this.
VAE slicing saves memory by splitting large batches of inputs into a single batch of data and separately processing them. This method works best when generating more than one image at a time, which will be the case usually, but keep in mind that inference will take longer the more images are generated in a run.
Generating images faster using Ays
I was researching how to speed up the inference of Stable Diffusion XL models on my GPU, and found this gem by Nvidia. And to my luck, its supported by Diffusers.
Now, how useful is using the AYS schedule for inference? In my experience, so-so.
Yes, it is much faster and does generate good enough images. But...the image quality in regards to prompt adherence is worse, and this depends alot on the model you are using. If it was fine tuned very well, it will perform mostly fine. If not, the images will most likely look weird.
Also (from my findings), it is best to use the sde-dpmsolver++ scheduler for this, due to it converging faster and giving good quality results (most of the time) in 10 steps of inference. Anything else will be less usable, but maybe i missed something.
I personally recommend using this to experiment with new ideas.
To get you started on using AYS with Diffusers, i give you another script.
#!./venv/bin/python3
import argparse
from pathlib import Path
import torch
from diffusers import (
DPMSolverMultistepScheduler,
StableDiffusionXLPipeline,
)
from diffusers.schedulers import AysSchedules
# Constants.
# ==========
IMAGES_BATCH_COUNT = 2
TORCH_DTYPE = torch.bfloat16
# Prepare and parse the terminal arguments.
# =========================================
args_parser = argparse.ArgumentParser(
description="Custom program for text-to-image using normal Stable Diffusion models."
)
args_parser.add_argument(
"--model",
type=str,
required=True,
help="Huggingface repository path to the SDXL model weights.",
)
args_parser.add_argument(
"--prompt-path",
type=Path,
default=Path("./prompt.txt"),
help="Filepath to the prompt on what to generate.",
)
args_parser.add_argument(
"--negative-prompt-path",
type=Path,
default=None,
help="Filepath to the prompt on what NOT to generate. Ignored if not set.",
)
args_parser.add_argument(
"--guidance-scale",
type=float,
default=3.0,
help="How close it should follow the prompt. Lower values result in more freedom, while higher ones result in more consistency.",
)
args_parser.add_argument(
"--images-count",
type=int,
default=IMAGES_BATCH_COUNT,
help="How many images should be generated per prompt.",
)
args_parser.add_argument(
"--seed",
type=int,
default=None,
help="The seed used to set the state of the PRNG. If not set, chooses a random one.",
)
args_parser.add_argument(
"--output-dirpath",
type=Path,
default=Path("./txt2img_batch"),
help="Where to store the generated images.",
)
parsed_args = args_parser.parse_args()
# Prepare the text-to-image pipeline to generate images.
# ======================================================
prompt = parsed_args.prompt_path.read_text()
negative_prompt = None if parsed_args.negative_prompt_path == None else parsed_args.negative_prompt_path.read_text()
pipeline = StableDiffusionXLPipeline.from_single_file(
parsed_args.model,
custom_pipeline="lpw_stable_diffusion_xl",
torch_dtype=TORCH_DTYPE,
safety_checker=None,
)
pipeline.scheduler = DPMSolverMultistepScheduler.from_config(
pipeline.scheduler.config,
algorithm_type="sde-dpmsolver++",
solver_order=2,
)
pipeline.enable_model_cpu_offload()
pipeline.enable_attention_slicing(slice_size="auto")
pipeline.vae.enable_slicing()
# Generate and save the images.
# =============================
batch_count, leftover_count = divmod(parsed_args.images_count, IMAGES_BATCH_COUNT)
images_batch_sequence = []
if batch_count > 0:
images_batch_sequence += [IMAGES_BATCH_COUNT] * batch_count
if leftover_count > 0:
images_batch_sequence += [leftover_count]
generator = None if parsed_args.seed == None else torch.manual_seed(parsed_args.seed)
parsed_args.output_dirpath.mkdir(
parents=True,
exist_ok=True,
)
for num_images_per_prompt in images_batch_sequence:
inference_result = pipeline(
prompt=prompt,
negative_prompt=negative_prompt,
guidance_scale=parsed_args.guidance_scale,
num_images_per_prompt=num_images_per_prompt,
generator=generator,
timesteps=AysSchedules["StableDiffusionXLTimesteps"],
)
file_counter = 0
for generated_image in inference_result.images:
output_filepath = parsed_args.output_dirpath.joinpath(f"image{file_counter}.png")
while output_filepath.exists():
file_counter += 1
output_filepath = parsed_args.output_dirpath.joinpath(f"image{file_counter}.png")
generated_image.save(output_filepath)
print(f"Saved the generated {num_images_per_prompt} image/s into the directory '{parsed_args.output_dirpath}'.")
Enjoy.
Conclusion
At the time of writing this article, it is doable and works OK-ish.
Software support (unless obtained from other sources) will get harder with time, sometimes forcing one to compile their own stuff to get it working. Now, why am i this opinion? cuML and cuDF not supporting Nvidia Pascal GPU's, but cupy for whatever reason working fine.
On to the topic of inference speed. It's slow, atleast on my GTX 1070. 2 image take normally around 3:40 minutes to generate with model offloading, and around 1:30 with AYS and model offloading. This is the fastest possible at the moment, atleast on a GPU as mine. I think it will be much faster on a GTX 1080ti, but these are not exacly budget cards.
If you ask me, buy a used Nvidia 2000 or 3000 series GPU. Best i can recommend is the RTX 3060 12 GB model, price to performance for model inference.