How To Perform Style Transfer With Stable Diffusion And Python

In this post, I will show you how to perform image style transfer with the help of a Stable Diffusion model. I will also use ControlNet and IP-Adapter for this task.

The Python code below can be executed in a Colab notebook. Make sure you have GPU enabled on your Colab notebook.

What is style transfer?

Style transfer means combining a base image with a style image so that the resultant image should contain the base image’s content and the style image’s style.

Consider the following two images.

stable diffusion style transfer python

Let’s say we want to add the style of image B to image A. The output image should look like this 👇

stable diffusion style transfer diffusers

Style transfer in Python

Install and import libraries

!pip install accelerate diffusers controlnet_aux
import torch
from controlnet_aux import CannyDetector
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from diffusers.utils import load_image, make_image_grid

Import ControlNet and Absolute Reality model

ControlNet model –

Absolute Reality model –

controlnet = ControlNetModel.from_pretrained(

pipe = StableDiffusionControlNetPipeline.from_pretrained(

Load IP Adapter



IP Adapter models –

# load style-image
ip_adap_img = load_image("painting.jpg")

#display image
ip adpater style transfer

This image will be used as the style image. The colors and the patterns in this image will be mixed with the content of the base image.

Download style image

Convert input image to Canny edge image

First, let’s load the base image.

# load input image
img = load_image("3d-person.jpeg").resize((768, 768))

stable diffusion style transfer

Download base image

Next, the base image will be converted into a Canny edge image.

# detect edges
canny = CannyDetector()
canny_img = canny(img, detect_resolution=512, image_resolution=768)

# display image
controlnet canny python

Perform style transfer

prompt = """
(photorealistic:1.2), raw, masterpiece, high quality, 
8k, girl wearing hoodie, headphones, hands in pocket

# weightage to style image

# generate images
images = pipe(prompt = prompt, 
              negative_prompt = "low quality",
              height = 768, 
              width = 768,
              ip_adapter_image = ip_adap_img,
              image = canny_img,
              guidance_scale = 6,
              controlnet_conditioning_scale = 0.7,
              num_inference_steps = 20,
              num_images_per_prompt = 3).images

This pipeline will generate 3 image variants.

You can check out my other blog posts on Stable Diffusion to learn more about the parameters used in the pipeline above.

The variable ‘images’ is a list that has 3 generated images. Let’s add the original base image to this list as well.

# insert the original image in the list of generated images
images = [img] + images

Display images

make_image_grid(images, cols=4, rows=1)
python style transfer AI

As you can see, the style image’s characteristics are visible in the generated images. This is due to the IP Adapter model.

Apart from that, all the generated images contain a 3d character of a girl wearing a hoodie and headphones. This was made possible by the ControlNet model.

Leave a Reply

Your email address will not be published. Required fields are marked *