Tutorial: Train SDXL LoRAs on DiffusionHub

Learn how to train SDXL LoRA models using DiffusionHub.io and bring your creative visions to life.

1. Getting familiar with Khoya SS GUI

Upon launching DiffusionHub, Choose Kohya instance in the dashboard. Familiarize yourself with the graphical user interface (GUI) to seamlessly navigate through the training process.

Note: For long training sessions, it is recommended to disable AFK protection.

By default, AFK protection is enabled and it will terminate the session if no activity is found for 30 mins.

(1) click on LORA tab to if you aim to train SDXL LoRA.

(2) Select the model you want to train with, we recommend SDXL base model(stable-diffusion-xl-base-1.0) which can be selected with a drop-down list. For any other model, you can select “custom” and then paste in the location of your model.

(3) Select the file type you want to train your LORAs on (Safetensors or CKPT). We recommend you use safetensors.

(4) Clicking on 4 will take you to the DiffusionHub file explorer, where you will set up your folders.

(5) Last is the logger, which will show you what’s going on in the background while you work in GUI.

2. Prepare your training images

More is better? Not necessarily

In our testing, we found that high-quality images give better results.

Training images can be as low as 15, the quality of images has a huge impact on your results.

SDXL supports training with high-resolution images (1024 X1024)

Variations in aspect ratios among images are acceptable. Enable the bucketing option during training to categorize images with different aspect ratios effectively.

Ensure that all selected images are either in PNG or JPEG format. If you are training a person, you better use more close-up photos/upper body photos and around 30% of your images should be full-body, if you want to train the full body.

Avoid below

a. Blurry

b. Similar backgrounds

c. Duplicate images

Cropping is not required but can give good results if the images have the same aspect ratio, you can use a free bulk image cropper (https://www.birme.net) to resize your images:

3. Navigating file browser and folder preparations.

Once you click on File Explorer, you will see two folders “output” and “data”

Click on the output folder

(6) Once you are inside the output folder, click on “add folder” to add three folders example, Output, Train_images and Regularization_images.

Note : Names can be as you wish.

(7) Output folder: This folder is where the kohya SS will prepare its folder structure.

(8) Train_images : This will contain the images you want to train LORA on. Upload your training images into this folder.

(9)Regularization images (Optional): Will contain the regularization images, upload your regularization images into this folder. it is recommended to have your regularization images in the same aspect ratios as your training images.

DiffusionHub offers a free 30-minute trial, which is a fantastic way to explore the platform’s capabilities without any commitment. This trial period allows you to familiarize yourself with the user interface, experiment with different models, and understand the potential of Stable Diffusion technology in image generation

4. Dataset preparations for training

Click on the “Dataset preparation” Tab and you will see below options

(10) Instance prompt: This is the unique identifier of your subject you want to train to make sure it is unique, we use “ohwx”, this will be used in your prompts to activate your LORA.

(11) Class: Will define the subject you want train, Ex: Man, Woman, Car, etc.

(12) Training images: This is where you will put the path of the folder where you have uploaded your training images in this case it is /output/Train_images

you can copy the folder path with the below icon showing on all directory.

Note: make sure you remove “/dh-user” from the copied path.

(13) Repeats: Will define how many times your training images will be used for training the LORA ( Aim for around 1600+ steps, so if I am using 15 training images make this 100) for 30 training images 55.

(14) Regularization images (Optional): Put the path of the regularization folder where you have put in the regularization images. You can skip this if you don’t want to use regularization images for training, ( these images are used to prevent overfitting and improve generalization performance).

(15) Repeats for Regularization images: Keep it 1

(16) The Destination training directory will be the path where Kohya SS will put its folder structure. In this case, it is /output/Output.

(17) Click on prepare training data: This will prepare the folder structure as per the inputs we have given. Click on the logger and you should see that the folder structure is created.

You should see 4 folders inside the output/Output folder: reg, img, log and model.

reg: contains your regularization images

img: contains your training images

log: training logs are collected

model: this will contain your SDXL Lora model after training.

(18) Click on copy info to folder tab: This will copy the Kohya SS folder structure to the folders tab automatically.

Name Your LORA: in the Model Output name, put your customized LORA name.

5. SDXL training parameters

If you are following along we recommend you train your LORAs as per the below training parameters and customize them as per your needs.

(19) LR Scheduler: COSINE and CONSTANT gave us the best results.

(20) Learing rate: 0.0004

(21) Optimizer: Adafactor

(22) Optimizer extra Arguments: “ scale_parameter=False relative_step=False warmup_init=False “

(23) Max resolution: 1024,1024

(24) Text encoder learning rate: 0.0004

(25) Unet learning rate: 0.0004

(26) Network Rank (Dimension): 32 or 16 (with a higher number your LORA file size will increase)

Click on advanced.

(28) Enable Gradient checkpointing

6. Captioning your images (Optional)

This will help SDXL to understand and train your images better.

(29) Click on “Utilities” tab and Select BLIP captioning, select the images folder in this case “/output/Output/img/55_Ohwx_woman“.

(30) Prefix to add to BLIP caption: use your instance prompt in this case “ohwx”

(31) Click caption images to generate captions for your images.

You should see txt files corresponding to each image which will contain the captions for each image, to see the caption you will have to download these txt files.

If you need to modify the captions as per your requirements then you will have to download them and edit and re-upload.

(32) Go back to LORA tab and click start training to start training your LORA

Keep checking your logs to see the progress of your training.

Note: Sometimes the logs will be frozen, don’t panic the training will be still in progress, you can click start training again multiple times to see the logs.

Once the training is done you will see the above logs on the logger and your model can be downloaded to your local machine from the folder /output/Output/model/<yourLORAname>

6. Upload and Use your trained LORAs in DiffusionHub Automatic 1111

You can upload the LORAs you created to Automatic 1111 in two ways.

Faster/Recommended : Use “Batchlinks downloader” extension (https://github.com/etherealxx/batchlinks-webui.git). If you see batchlinks downloader in GUI then proceed to the next step, if not then you can download it as explained below.

Go to Extensions > Install from URL > put the git URL for Batch link downloader https://github.com/etherealxx/batchlinks-webui.git > click install and once the installation is complete, reload the UI.

Upload your LORA on any of your platforms, Gdrive ,Hugging face etc.. And use the Batchlinks downloader extension to download any model or LORA using your link.

For the current training demonstration put (as shown below) #lora and the LORA link, which will send the LORA file directly to Automatic 1111 LORA directory.

Another way (Slower compared to the last method ) is to upload your LORAs to the Automatic 1111 file explorer manually from your local machine to the LORA folder in Automatic 1111 file Explorer.

The path to the LORA folder is /data/models/Lora , click upload files and upload your LORA.

After refreshing you should also see LORA in your Auto 1111 GUI

NOW YOU ARE ALL SET TO TEST YOUR LORA!

Here are some of the results we were able to get from our training

With a user-friendly interface and a free 30-minute trial, DiffusionHub is the perfect place to start your journey into the future of visual content creation. Embrace the potential of AI image generation technology with DiffusionHub today!