Add example script for training/fine-tuning ELLA

[budui]

Add example script for training/fine-tuning ELLA

TODO

• Support fine-tuning ELLA itself
• Support fixed ELLA, fine-tuned LoRA
• Support ELLA+CLIP as condition at the same time during training

[budui]

[1/3] Train text to image lora with ella.

Modified from https://huggingface.co/docs/diffusers/training/lora#launch-the-script

export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export OUTPUT_DIR="/tmp/finetune_lora_pokemon"
export DATASET_NAME="diffusers/pokemon-llava-captions"
# export DATASET_NAME="/path/to/local/dataset/huggingface/datasets/diffusers___parquet/diffusers--pokemon-llava-captions-d6a88a772327b199/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec"

accelerate launch --mixed_precision="fp16"  train_text_to_image_lora_with_ella.py \
 --pretrained_ella /path/to/the/pretrained/ella-sd1.5-tsc-t5xl.safetensors \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --dataset_name=$DATASET_NAME \
  --dataloader_num_workers=8 \
  --resolution=512 \
  --center_crop \
  --random_flip \
  --train_batch_size=1 \
  --mixed_precision="fp16" \
  --gradient_accumulation_steps=4 \
  --max_train_steps=15000 \
  --learning_rate=1e-04 \
  --max_grad_norm=1 \
  --lr_scheduler="cosine" \
  --lr_warmup_steps=0 \
  --output_dir=${OUTPUT_DIR} \
  --checkpointing_steps=500 \
  --seed=1337

[XIONGPEILIN]

我尝试了一下使用interaction diffusion的代码里面加入ella进行训练，https://github.com/XIONGPEILIN/ella_with_interactiondiffusion-
感觉是哪里出了问题，没有训练成功