Tutorial: How to Set Up ControlNet in Stable Diffusion Web UI

So maybe you’ve heard about ControlNet, or maybe you haven’t but have seen some of the truly amazing images that it has been able to achieve, but what is it? How can you set up ControlNet and start using it yourself in Stable Diffusion? Well luckily this guide is here to help you get started!

What is ControlNet and why use it?

ControlNet takes the standard img2img tool in Stable Diffusion and ratchets it up to 11. Although img2img can truly transform your input images into amazing outputs (see my article about using img2img), you ultimately can’t control minute features or the specific poses of your characters. For example, here is an example of me using a combination of img2img, in-painting and Photoshop to take an image of Sleeping Beauty from the internet and turn it into something new for use in my illustrated short story:

While in the above I intentionally changed many details to significantly transform the output into its own original picture, let’s say you want to keep all of the original features, composition and pose of an image down to the most minute details of the face, posture and background details, but convert it into a photorealistic version, such as this set of images produced by /u/All_I_Can and posted along with their ControlNet workflow on Reddit:

Pretty impressive stuff! And these are the ControlNet models that /u/All_I_Can used to create this image:

This is a major milestone for users to actually be able to fine tune, control and replicate images to a degree previously unattainable through Stable Diffusion’s settings. So how can you get in on this?

Prerequisites before getting started with ControlNet

Before you can use ControlNet in Stable Diffusion, you need to actually have the Stable Diffusion WebUI. If you don’t already have Stable Diffusion, there are two general ways you can do this:

Option 1: Download AUTOMATIC1111’s Stable Diffusion WebUI by following the instructions for your GPU and platform…

• Installation instructions for Windows (NVIDIA GPU)
• Installation instructions for Windows (AMD GPU)
• Installation for Linux

Option 2: Use a pre-made template of Stable Diffusion WebUI on a configurable online service. I’ve written an article comparing different services and the advantages of using Stable Diffusion AUTOMATIC1111 v1.5 and v2.1 on RunPod.io. This tutorial and its screenshots were taken using RunPod.io, but the process still applies if you have the SD WebUI on your local hardware as well.

Also be aware that while ControlNet models will work with the base StableDiffusion model, there are many custom trained models out there such as DreamLike PhotoReal that you will need to download and install separately and in addition to ControlNet.

Getting the ControlNet Models

Regardless of whether you have the Stable Diffusion WebUI on your own computer or are using a web service like RunPod.io, you’re going to need to download the ControlNet models. There are quite a few different models, so if you are unsure which ones you want, you can try them for free first at https://huggingface.co/spaces/hysts/ControlNet.

If you want to pick and choose specific models to download, you can get their *.safetensor files here: https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main. Each individual file is just over 700MB, so this may take a while.

Installing ControlNet in Stable Diffusion

So you have the Stable Diffusion WebUI and you have downloaded the ControlNet models. Great! Run StableDiffusion in your browser, then navigate to Extensions. There are two ways you can get ControlNet here.

Option 1:

If you have Stable Diffusion on your own platform, copy and paste the following address into the text field under the Install from URL tab:

https://github.com/Mikubill/sd-webui-controlnet

After installing you will want to close the WebUI and restart it.

Option 2:

If you’re like me and are using RunPod.io instead of running Stable Diffusion on your own platform, then Option 1 will still work, but you also have an expedited option for getting ControlNet. Under Extensions, go to the Available tab:

Now scroll down the list of available extensions until you find the one for sd-webui-controlnet manipulations and OpenPose Editor tab (if you want to use the OpenPose model). Hit the Install button for both options

After this, go to the Settings tab and hit the big Reload UI button in the upper right corner. In RunPod.io it is possible that you will get a Bad Gateway error. If this is the case, fear not! Simply wait a moment and then hit the Refresh button in the browser tab you’re using and the Stable Diffusion WebUI will come back.

Add the ControlNet Models to the Stable Diffusion WebUI folder

If you went the route of downloading each individual *.safetensor file from https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main, then your life is easy. Simply place those files in the stable-diffusion-webui > extensions > sd-webui-controlnet > models directory, regardless of whether that is on your own platform or in the JupyterLab Notebook of RunPod.io.

However, if you just want to download all the model files together or you want to transfer all files directly to RunPod.io from Hugging Face without ever downloading them yourself, this is how you do it:

Open up Terminal and direct it to the directory where the file (downloaded from https://github.com/git-lfs/git-lfs/releases/download/v2.9.0/git-lfs-linux-amd64-v2.9.0.tar.gz) is located and where you want to unpack it. In Terminal, paste and enter:

tar -xf git-lfs-linux-amd64-v2.9.0.tar.gz

Once the file unpacks and Terminal returns, paste and enter:

chmod 755 install.sh

Once Terminal returns, paste and enter:

./install.sh

Direct Terminal to stable-diffusion-webui > extensions > sd-webui-controlnet > models, then paste and enter:

git clone https://huggingface.co/webui/ControlNet-modules-safetensors

It will now download all Model files directly from Hugging Face into the ControlNet folder.

Enable Multi ControlNet in Stable Diffusion Web UI

Now go back to the Stable Diffusion WebUI in your browser. Under Settings, go to ControlNet and then select the number of models that you want to be able to apply to a generation simultaneously. In the example at the very top of this tutorial, /u/All_I_Can applied three simultaneous models (OpenPose, Depth and HED) to generate the hyper realistic image of the woman in Pompeii. This is called Multi ControlNet. Otherwise, the default setting for ControlNet is just one model amount.

Once you are done setting up multi-ControlNet, hit the Apply Settings button. In RunPod.io, you may see a Bad Gateway error, in which case, simply refresh the tab in your browser.

Start Using ControlNet in the Stable Diffusion Web UI

ControlNet should now appear as a new section under the text2img and img2img tabs of the Stable Diffusion WebUI. Click the downward facing arrow to expand it. For each model you want to apply in your generation, make sure that you have the Enable box checked, and have both the PreProcessor and Model menus selected for the specific model you want to apply. You will need to do this for each individual ControlNet model you wish to apply simultaneously in your image generation.

If you decide to add additional model *.safetensor files to the ControlNet directory after initial set up, then hit the blue refresh button on the right to get them to show up.

And now your set-up is complete! You can start playing with different models by uploading an image of your choice under the ControlNet section of the WebUI to generate the control image, and then setting up your prompts, sampling method and parameters are you normally would using the Stable Diffusion WebUI. Have fun!

Related Articles and Tutorials