🚗 Preamble 🚗
This tutorial will guide you on how to create your own LoRA model from scratch. Even if you don’t have a high-performance computer, you can complete the entire process using free tools like Google Colab and Google Drive. The tutorial focuses mainly on anime-style creations but is also applicable to realistic styles. Please note that copying real human faces without authorization is not supported.
👐Making a LoRA
What is LoRA?
● LoRA is a lightweight model training technique that can customize the generation of:Characters or individuals
● Art styles
● Poses or concepts
Common Types of LoRA
1. LoRA (Classic): Suitable for most use cases.
2. LoCon: Designed for in-depth learning of artistic styles with more layers.
3. LoHa/LoKR (IA)^3: A new algorithm with a more limited scope.
📊 First Half: Creating a Dataset
A dataset is the core of making a LoRA. A high-quality dataset significantly improves the generated model's results. A dataset consists of images and their descriptions, where each image must have a corresponding text file with the same filename (e.g., 1.png and 1.txt).
Dataset Quality Requirements
1. Diversity of features: Include various poses, angles, backgrounds, and outfits.
2. Recommended image count: Minimum of 5 images, ideally 20+, up to a maximum of 1000.
3. Descriptive content: Ensure descriptions are clear and specific. For example:
General style: "A full-body photo of a blonde girl sitting on a chair."
Anime content: Use booru tags, such as 1girl, blonde hair, full body, on chair
Dataset Creation Process
1️⃣ Set Up the Project
● Open Dataset Maker Notebook and connect to Google Drive.
● Set the project name and folder structure.
2️⃣ Collect Images
● If you don’t have images, search for suitable ones on platforms like Gelbooru based on your desired character or style.
● Gelbooru’s booru tags will help automate the description process later.
3️⃣ Filter Images
● Use FiftyOne AI to detect and remove duplicate images and clean up low-quality or irrelevant ones.
4️⃣ Generate Descriptions
● Use WD 1.4 Tag AI for anime images or BLIP AI for realistic images to automatically generate descriptions.
5️⃣ Optimize Tags (Optional)
● Specify an activation tag (e.g., project_name) as a trigger word for easy use later.
● Merge redundant tags, e.g., combining striped shirt and vertical stripes into striped shirt.
6️⃣ Complete the Dataset
● The dataset will be stored in the designated Google Drive folder, ready for training.
📖 Second Half: Training a LoRA
Training a LoRA is a more technically demanding step, but tools like LoRA Trainer Notebook or LoRA Trainer XL Notebook simplify the process.
Key Settings
● Basic Settings
○ The project name must match the one set during the dataset creation phase.
○ Choose a base model like SD 1.5 or SDXL, or import a custom model.
● Dataset Processing
○ Resolution: 512 is the recommended default. For more detail, increase the resolution, but note it will extend training time.
○ Tag Randomization (shuffle_tags): Enable for anime content to avoid overfitting.
○ Activation Tags: Enable this option if you’ve set trigger words.
● Training Steps
○ Total steps depend on:Image Count × Repetitions × Epochs ÷ Batch Size
○ Recommended setups:20 images × 10 repetitions × 10 epochs ÷ 2 batch size = 1000 steps.
■ 100 images × 3 repetitions × 10 epochs ÷ 2 batch size = 1500 steps.
● Learning Rates
○ Unet Learning Rate: Recommended at 5e-4 for beginners. For more stable training, set to 1e-4 or 2e-4.
○ Text Encoder Learning Rate: Usually set to 1/2 or 1/5 of the Unet rate, e.g., 1e-4 or 5e-5.
Starting the Training
1. Open the LoRA Trainer Notebook.
2. Enter the project name, dataset path, and base model path.
3. Set training parameters and run the code. Once the training is complete, the results will automatically be saved to Google Drive.
🎨 Third Part: Testing the LoRA
After training, you need to test the model to ensure it meets your requirements.
Testing Process
1. Download the Model File
a. The training results are stored in the /lora_training/outputs folder in Google Drive. Each file corresponds to a different training stage (e.g., 01, 02, 03, etc.).
2. Use the LoRA in Prompts
a. Add the trained LoRA model to your prompts and adjust the weight (e.g., 0.7 or 1.0).
3. Compare Training Stages
a. Use WebUI’s drawing feature to compare the effects of different epochs and find the best version.
4. Adjust Weights
a. Test different weights (e.g., :0.5, :0.6, :0.7) and choose the value that generates the best results.
By following this process, you can create, train, and test your own LoRA models easily and efficiently!
