In this tutorial, I will teach you how to generate images by merging multiple LoRA models in Python. I will use the Diffusers library and SDXL 1.0 base model for this task.
I will write the Python code in a Kaggle notebook to implement AI image generation using multiple LoRA models because Kaggle provides 30 hours of free GPU usage every week. To run models like SDXL 1.0, you must have a decent Cuda-enabled GPU.
The LoRA models for this tutorial will be downloaded from CivitAI where you can find plenty of LoRA models as well as other image generation model checkpoints.
Open Kaggle Notebook
Login to your Kaggle account and verify your account by providing your phone number. Once your account is verified you will be able to access the GPU in Kaggle notebooks.
Click on the ‘Create’ button in the left sidebar of your Kaggle account and then click on the ‘New Notebook’ option to open a Kaggle notebook.
Inside the Kaggle notebook, check out the sidebar on the right side and expand the ‘Notebook options’ menu. Select GPU t4 x2 as the Accelerator.
After that, you can run the code for AI image generation.
Install Diffusers and PEFT
PEFT (Parameter-Efficient Fine-Tuning) is a library that is used for adapting pre-trained language models to various tasks without fine-tuning all the model’s parameters.
Here PEFT is helpful in managing multiple LoRA models for generating unique images. So, in the top most cell of the Kaggle notebook, run the following commands.
!pip install diffusers
!pip install peft
Download SDXL 1.0 model
Now let’s download SDXL 1.0 model from Hugging Face. We will use the LoRA models with SDXL 1.0 as the base model.
from diffusers import DiffusionPipeline
import torch
pipe_id = "stabilityai/stable-diffusion-xl-base-1.0"
pipe = DiffusionPipeline.from_pretrained(pipe_id,
torch_dtype=torch.float16).to("cuda")
Generate an image using SDXL 1.0
Let’s generate an image using our base model. We will use the same prompt throughout this article.
prompt = "hiker, beautiful forest"
image = pipe(
prompt, num_inference_steps=28,
generator=torch.manual_seed(0) # seed to reproduce same output
).images[0]
# display image
image
Use Papercut SDXL LoRA
Now we will download a LoRA model known as Papercut SDXL and try to generate another image using the same prompt.
Download Papercut SDXL LoRA
We can directly download any model from CivitAI in our Kaggle notebook. Simply use the command below.
!wget https://civitai.com/api/download/models/133503 --content-disposition
You can use the same command to download LoRA models from CivitAI in Google Colab as well.
Load LoRA weights
Let’s integrate the Papercut SDXL model weights with the SDXL 1.0 pipeline. The downloaded LoRA model will be saved in /kaggle/working/ directory.
pipe.load_lora_weights(pretrained_model_name_or_path_or_dict="/kaggle/working/papercut.safetensors", adapter_name="papercut")
If you face any issue in downloading LoRA models from CivitAI, then you can also find LoRA models at Hugging Face. Simply replace the pretrained_model_name_or_path_or_dict parameter by the model ID from Hugging Face as shown below.
pipe.load_lora_weights(pretrained_model_name_or_path_or_dict="TheLastBen/Papercut_SDXL", adapter_name="papercut")
Generate image using Papercut SDXL LoRA
There are certain trigger words that are usually associated with LoRA models. These trigger words or trigger tags are specified during training of the LoRA models. For example the trigger word for Papercut SDXL LoRA model is ‘papercut’.
So, to generate the image using this LoRA model, simply add the trigger word ‘papercut’ at the beginning of the prompt.
prompt = "papercut, hiker, beautiful forest"
# regulate contribution of LoRA model in image generation
lora_scale= 0.9
image = pipe(
prompt, num_inference_steps=28,
cross_attention_kwargs={"scale": lora_scale},
generator=torch.manual_seed(0)
).images[0]
# display image
image
Use Moebius (Jean Giraud) Style LoRA
After using Papercut SDXL LoRA, let’s download another LoRA model from CivitAI and generate a similar image. This time I am going to use the Moebius (Jean Giraud) Style LoRA model.
Download Moebius (Jean Giraud) Style LoRA
!wget https://civitai.com/api/download/models/154071 --content-disposition
Load LoRA weights
pipe.load_lora_weights(pretrained_model_name_or_path_or_dict="/kaggle/working/Moebius (Jean Giraud) Style.safetensors", adapter_name="moebius")
# enable LoRA model
pipe.set_adapters("moebius")
Generate image using Moebius (Jean Giraud) Style LoRA
To generate image with this new LoRA model, we will again have to change the image prompt by including a new trigger word. The trigger word for this LoRA model is ‘Moebius (Jean Giraud) Style page’.
prompt = "Moebius (Jean Giraud) Style page, hiker, beautiful forest"
lora_scale= 0.9
image = pipe(
prompt, num_inference_steps=28,
cross_attention_kwargs={"scale": lora_scale},
generator=torch.manual_seed(0)
).images[0]
image
As you can see, the style of the generated is drastically different from that of the Papercut SDXL LoRA model. Now we will combine both Papercut SDXL and Moebius (Jean Giraud) Style LoRA models and generate a new image using the same prompt.
Combine multiple LoRA models
Let’s define a weighted combination of the two LoRA models that we have used in this tutorial.
pipe.set_adapters(["papercut", "moebius"], adapter_weights=[1.0, 0.5])
Weight given to Papercut SDXL model is 1.0 and weight given to Moebius (Jean Giraud) Style model is 0.5. You can change the weights as per your requirements.
Generate an image using multiple LoRA models
Since I am going to two multiple LoRA models with the SDXL 1.0 base model, I will have to include the trigger words of all the LoRA models in my prompt.
prompt = "papercut, Moebius (Jean Giraud) Style page, hiker, beautiful forest"
image = pipe(
prompt, num_inference_steps=28,
cross_attention_kwargs={"scale": 0.9},
generator=torch.manual_seed(0)
).images[0]
image
Awesome! The output image contains styles from two LoRA models. Similarly, you can use more than two LoRA models and create a variety of images with unique styles.
