In the previous article we used Stable Diffusion and ControlNet to transform a logo into a natural looking image.
But now, let’s push the boundaries even further. Imagine you have a picture, and you wish you could replace something in it with a different object. Enter Stable Diffusion once again, your ally in this creative journey.
Think of it like this: you have a puzzle, and one piece is missing. Or you have a portrait that’s almost perfect, just missing a key detail. Can Stable Diffusion help us solve these artistic challenges?
Absolutely! There’s a special type of AI models known as inpainting models. They excel at not only fixing images but also enhance the images using the magic of Generative AI.
So, in this article, we’re going to explore the capabilities of Stable Diffusion Inpainting for image enhancement. We’ll learn how to use it to replace objects in images, giving your creativity a whole new canvas to work on. Let’s dive in and discover the art of AI-powered image editing and enhancement together!
What is inpainting?
In the context of image generation, inpainting can be defined as using a stable diffusion model to generate certain selected parts of a given image. Refer the illustration below to get more clarity.
As you can see, the user firsts select the region in the input image and then the stable diffusion inpainting model would generate new items in that selected region based on a given prompt.
Implement Stable Diffusion Inpainting using Diffusers
We will use Google Colab with GPU enabled to run this code. Feel free to download the Python notebook right away and run the code at your end by clicking the button below.
First let’s install the required packages to implement Stable Diffusion inpainting.
!pip install diffusers["torch"] transformers gradio
Now we will import the libraries and modules.
import os
import torch
import numpy as np
import gradio as gr
from PIL import Image
from diffusers import StableDiffusionInpaintPipeline
Next we will create the Stable Diffusion pipeline to perform inpainting. The inpainting Stable Diffusion model will be downloaded from Hugging Face.
pipeline = StableDiffusionInpaintPipeline.from_pretrained(
"runwayml/stable-diffusion-inpainting",
torch_dtype=torch.float16,
)
pipeline = pipeline.to("cuda")
Let’s create a function that will perform inpainting on an input image and it will also use a prompt that will help in adding new elements to the image.
You may change the prompt inside the function or continue with this only.
def predict(mask_img):
prompt = "a green frog, highly detailed, natural lighting"
image = pipeline(prompt=prompt,
num_inference_steps=35,
image=mask_img["image"],
mask_image=mask_img["mask"],
guidance_scale=9).images[0]
return image
To select the areas on the input image we need an interactive user interface that should allow us to upload an image and select the regions of interest using a pen or a brush. This is where Gradio comes into the picture. It is a user friendly and intuitive tool to create simple graphical user interface using Python.
Run the code below to launch the Gradio UI.
gr.Interface(
predict,
title = 'Stable Diffusion In-Painting Tool on Colab with Gradio',
inputs=[
gr.Image(source = 'upload', tool = 'sketch', type = 'pil')
],
outputs = [
gr.Image()
]
).launch()
Let’s upload an image to this interface and next we will use the brush tool to select the region that we want to edit.
As you can see, the person in the image has been selected with the brush tool. Now we can click on the Submit button to generate a new object in the selected region.
Awesome! We have a giant frog in place of a man. You can further fine-tune the image to improve the quality of the generation. We can also perfrom inpainting using Automatic1111 platform that requires no coding. Check out how to setup Automatic1111 and run on a cloud instance.
