This article is a guide explaining how to train LoRA (Low-Rank Adaptation) using the Illustrious model based on Stable Diffusion XL (SDXL). This guide is intended for users with some knowledge and focuses on the training process to fine-tune specific characters or styles, leveraging the characteristics of the Illustrious model, which specializes in generating anime and illustrations. The following sections provide detailed explanations and explore future prospects.
1. Introduction to LoRA and Its Benefits
LoRA is a technology for efficiently adjusting large language models and diffusion models. In the context of Stable Diffusion, it allows you to specialize in specific tasks (e.g., generating specific characters or styles) by training a small number of additional parameters. When combined with the Illustrious model, this technology enhances flexibility and efficiency in anime-style image generation. Research shows that LoRA significantly reduces storage requirements and shortens training time (LoRA - Hugging Face).
2. Understanding the Illustrious Model
The Illustrious model is an SDXL-based model developed by OnomaAI Research and trained on the Danbooru dataset (Illustrious: an Open Advanced Illustration Model - Arxiv). Version 0.1 is an unadjusted base model with limited resolution and safety controls compared to later versions, but it serves as a foundation for LoRA training. The strength of this model lies in its rich character knowledge and ability to generate anime styles, making it suitable for adding specific details through LoRA training.
3. Dataset Preparation
Dataset preparation for LoRA training varies depending on the training target (e.g., character, style). For character LoRA, collect 20-40 images of the target character and assign appropriate tags (e.g., hair color, clothing) to each image. For style LoRA, collect images in a specific artistic style (e.g., pixel art) and add tags for background and lighting. For the Illustrious model, it is recommended to prioritize SFW images and avoid NSFW images (How to train Pony/Illustrious lora with multiple costumes | Civitai).
4. Selection of Training Parameters
Training parameters significantly impact LoRA training with the Illustrious model. The following table shows the recommended parameter ranges:
| Parameter | Recommended Range | Notes |
| UNET Learning Rate | 0.0003 - 0.0005 | Adjusts the strength of character features |
| Text Encoder Learning Rate | 0.00003 - 0.00005 | Typically 1/10 of UNET |
| Number of Epochs | 10 - 20 | Adjust according to dataset size |
| Network Dimension (Dim) | 64 | Balance between file size and level of detail |
| Alpha | 32 | Typically half of Dim |
These values need to be adjusted according to the characteristics of the Illustrious model (e.g., anime style generation) (Model Training - Illustrious NoobAI LoRA Discussion | Tensor.Art).
5. Training the LoRA Model
The training process is performed using SeaArt model training. First, select the Illustrious model as the base, then set up the dataset and parameters. During training, monitor the logs and check for signs of overfitting or underfitting (e.g., image saturation, increased noise). Training is typically completed in 1500-3000 steps, and it is recommended to save epochs periodically (How to train Lora models - Stable Diffusion Art). With the Illustrious model, convergence occurs at around 3000 steps.
6. Model Evaluation and Fine-tuning
After training, test the generated LoRA model and evaluate the expected results (e.g., reproduction accuracy of specific characters, style matching). If the results are insufficient, adjust parameters (e.g., learning rate, number of epochs) and train again.
7. Evolution from v0.1
Illustrious model v0.1 was trained at a resolution of 1024×1024 and functions as a basic base model. On the other hand, v1.0 achieves a maximum resolution of 1536×1536 and integrates natural language processing and tag-based prompt processing (Illustrious XL 1.0 - v1.0 | Illustrious Checkpoint | Civitai). This enables v1.0 and v1.1 to generate more detailed and clearer images, and also improves LoRA compatibility. The limitations of v0.1 (e.g., noisiness, instability in art style) have been improved in later versions (Illustrious-XL - としあきdiffusion Wiki).
8. Comparison with Other Models
Comparing the Illustrious model with Stable Diffusion 1.5 and other XL models (e.g., Pony), Illustrious has rich knowledge of anime characters and is suitable for LoRA training. However, Stable Diffusion 1.5 is more suitable for general purposes and is not as specialized in specific styles as Illustrious. Compared to the Pony model, Illustrious may achieve good results with a smaller dataset, but parameter settings may need to be adjusted (How to train Pony/Illustrious lora with multiple costumes | Civitai).
9. Future Prospects
The future of LoRA training for the Illustrious model depends on advances in AI and machine learning. Higher resolution, more efficient training algorithms, and community contributions (e.g., development of LoRA and ControlNet) are expected. As of March 2025, the activities of the open-source community are supporting model evolution, and customization possibilities are likely to expand further in the future (OnomaAIResearch/Illustrious-xl-early-release-v0 · Hugging Face).
