2024.05.14 Diffusion 代碼學習筆記

配環境

我個人用的是Geowizard的環境：https://github.com/fuxiao0719/GeoWizard。

出于方便考慮，用的pytorch官方的docker容器，因此python版本（3.10）和原作者（3.9）不同，其余都是一樣的。

https://hub.docker.com/r/pytorch/pytorch/tags?page=&page_size=&ordering=&name=2.0.1

安裝的時候，把requirements.txt 里面開頭的torch和torchvision刪掉就行了（因為我們的docker容器里已經有了，版本也是一樣的）。

如果遇到以下兩個報錯可以這樣解決：
apt install libgl1-mesa-glx # ImportError: libGL.so.1: cannot open shared object file: No such file or dir
apt-get install libxrender1 # ImportError: libXrender.so.1: cannot open shared object file: No such file or directory

hugging face的下載問題

把原網站換成鏡像，不然下不下來。
pip install -U huggingface_hub
export HF_ENDPOINT=https://hf-mirror.com

入門學習

網站

就用這個網站的tutorial就行：
https://huggingface.co/docs/diffusers/tutorials

打不開的話看這個鏡像：把huggingface.co換成hf-mirror.com即可
https://hf-mirror.com/docs/diffusers/tutorials/autopipeline
注意，沒有魔法的話，鏡像中還是無法顯示圖片。但代碼文字都有，所以也可以將就著用了。

學習內容

我看的2024.05.15版本是這些內容：
在這里插入圖片描述

學習建議

我先開始用的還是pycharm。后來感覺還是jupyter notebook方便一點。
每個教程可能會用到不同的stable diffusion模型，比如"stabilityai/stable-diffusion-xl-base-1.0"。每次下載模型都很大，費時且費存儲空間。
- 如果只是學習的話，可以試試都用““CompVis/stable-diffusion-v1-4”，如果報錯了再試試"runwayml/stable-diffusion-v1-5"，實在不行再換成教程里的。這樣就不用老下載新模型了
建議先看好所有要學的教程，把里面的模型都load完，然后再一個個仔細看，因為模型下載真的很費時

學習總結

diffusers的用法

簡單的說，可以直接使用一個pipeline。也可以用scheduler和model。
model一般包括一個最基本的UNet（輸入current xt和t，輸出predicted noise）
- 如果是latent diffusion的話還有VAE
- 如果是text conditioned的話還有tokenlizer和text_encoder
有的scheduler有noise scaler參數，比如這位：
https://huggingface.co/docs/diffusers/v0.27.2/en/api/schedulers/unipc#diffusers.UniPCMultistepScheduler
AutoPipeline就是，指定模型（比如runwayml/stable-diffusion-v1-5）和任務（比如text2img, img2img, 或inpainting），自動給你找到那個pipeline對應的類
noise scheduler有一個add_noise方法，大概是下面這樣：

其他

這個過程中會下載很多模型或數據集，默認的路徑是：
~/.cache/huggingface/hub
可能需要pip install datasets, 下載的路徑還是~/.cache/
如果還沒研究明白怎么把模型上傳到huggingface，可以先把training config里面的push_to_hub設置為False
jupyter里面的PIL Image可以直接顯示（運行代碼image即可）；pycharm的話可以image.save(‘temp.png’), 然后看這個圖

Understanding pipelines, models and schedulers

代碼例子1：直接使用DDPM的pipeline

from diffusers import DDPMPipelineddpm = DDPMPipeline.from_pretrained("google/ddpm-cat-256", use_safetensors=True).to("cuda")
image = ddpm(num_inference_steps=25).images[0]
image

代碼例子2：分解DDPM的pipeline：先load scheduler 和model，然后使用

from diffusers import DDPMScheduler, UNet2DModelscheduler = DDPMScheduler.from_pretrained("google/ddpm-cat-256")
model = UNet2DModel.from_pretrained("google/ddpm-cat-256", use_safetensors=True).to("cuda")
scheduler.set_timesteps(50)import torchsample_size = model.config.sample_size
noise = torch.randn((1, 3, sample_size, sample_size), device="cuda")input = noisefor t in scheduler.timesteps:with torch.no_grad():noisy_residual = model(input, t).sampleprevious_noisy_sample = scheduler.step(noisy_residual, t, input).prev_sampleinput = previous_noisy_sample
from PIL import Image
import numpy as npimage = (input / 2 + 0.5).clamp(0, 1).squeeze()
image = (image.permute(1, 2, 0) * 255).round().to(torch.uint8).cpu().numpy()
image = Image.fromarray(image)
image

代碼例子2：stable diffusion的inference

https://hf-mirror.com/docs/diffusers/using-diffusers/write_own_pipeline

# https://huggingface.co/docs/diffusers/using-diffusers/write_own_pipelinefrom PIL import Image
import torch
from transformers import CLIPTextModel, CLIPTokenizer
from diffusers import AutoencoderKL, UNet2DConditionModel, PNDMScheduler'''Load Stuff'''
vae = AutoencoderKL.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="vae", use_safetensors=True)
tokenizer = CLIPTokenizer.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="tokenizer")
text_encoder = CLIPTextModel.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="text_encoder", use_safetensors=True
)
unet = UNet2DConditionModel.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="unet", use_safetensors=True
)from diffusers import UniPCMultistepSchedulerscheduler = UniPCMultistepScheduler.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="scheduler")torch_device = "cuda"
vae.to(torch_device)
text_encoder.to(torch_device)
unet.to(torch_device)'''Prepare and process input text'''
prompt = ["a photograph of an astronaut riding a horse"]
# prompt = ["a photograph of the cartoon character SpongeBob SqurePants"]
height = 512  # default height of Stable Diffusion
width = 512  # default width of Stable Diffusion
num_inference_steps = 25  # Number of denoising steps
guidance_scale = 7.5  # Scale for classifier-free guidance
generator = torch.manual_seed(1)  # Seed generator to create the initial latent noise
batch_size = len(prompt)text_input = tokenizer(prompt, padding="max_length", max_length=tokenizer.model_max_length, truncation=True, return_tensors="pt"
)with torch.no_grad():text_embeddings = text_encoder(text_input.input_ids.to(torch_device))[0]max_length = text_input.input_ids.shape[-1]
uncond_input = tokenizer([""] * batch_size, padding="max_length", max_length=max_length, return_tensors="pt")
uncond_embeddings = text_encoder(uncond_input.input_ids.to(torch_device))[0]text_embeddings = torch.cat([uncond_embeddings, text_embeddings])'''Diffuse'''
latents = torch.randn((batch_size, unet.config.in_channels, height // 8, width // 8),generator=generator
).to(torch_device)latents = latents * scheduler.init_noise_sigmafrom tqdm.auto import tqdmscheduler.set_timesteps(num_inference_steps)for t in tqdm(scheduler.timesteps):# expand the latents if we are doing classifier-free guidance to avoid doing two forward passes.latent_model_input = torch.cat([latents] * 2)latent_model_input = scheduler.scale_model_input(latent_model_input, timestep=t)# predict the noise residualwith torch.no_grad():noise_pred = unet(latent_model_input, t, encoder_hidden_states=text_embeddings).sample# perform guidancenoise_pred_uncond, noise_pred_text = noise_pred.chunk(2)noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)# compute the previous noisy sample x_t -> x_t-1latents = scheduler.step(noise_pred, t, latents).prev_sample'''Decode the image: latent to rgb'''
# scale and decode the image latents with vae
latents = 1 / 0.18215 * latents
with torch.no_grad():image = vae.decode(latents).sample
image = (image / 2 + 0.5).clamp(0, 1).squeeze()
image = (image.permute(1, 2, 0) * 255).to(torch.uint8).cpu().numpy()
image = Image.fromarray(image)
image.save('temp.png')

結果

下面是我seed為0和1分別生成的結果：
prompt：a photograph of an astronaut riding a horse
在這里插入圖片描述

AutoPipeline

task = 'img2img' # text2img, img2img, inpainting
if task == 'text2img':from diffusers import AutoPipelineForText2Imageimport torchpipeline = AutoPipelineForText2Image.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16, use_safetensors=True).to("cuda")prompt = "peasant and dragon combat, wood cutting style, viking era, bevel with rune"image = pipeline(prompt, num_inference_steps=25).images[0]image.save('temp.png')if task == 'img2img':from diffusers import AutoPipelineForImage2Imageimport torchimport requestsfrom PIL import Imagefrom io import BytesIOpipeline = AutoPipelineForImage2Image.from_pretrained("runwayml/stable-diffusion-v1-5",torch_dtype=torch.float16,use_safetensors=True,).to("cuda")prompt = "a portrait of a dog wearing a pearl earring"url = "https://upload.wikimedia.org/wikipedia/commons/thumb/0/0f/1665_Girl_with_a_Pearl_Earring.jpg/800px-1665_Girl_with_a_Pearl_Earring.jpg"response = requests.get(url)image = Image.open(BytesIO(response.content)).convert("RGB")image.thumbnail((768, 768))image.save('1665_Girl_with_a_Pearl_Earring.png')image = pipeline(prompt, image, num_inference_steps=200, strength=0.75, guidance_scale=10.5).images[0]image.save('temp.png')if task == 'inpainting':from diffusers import AutoPipelineForInpaintingfrom diffusers.utils import load_imageimport torchpipeline = AutoPipelineForInpainting.from_pretrained(# "stabilityai/stable-diffusion-xl-base-1.0","runwayml/stable-diffusion-v1-5",torch_dtype=torch.float16, use_safetensors=True).to("cuda")img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"init_image = load_image(img_url).convert("RGB")mask_image = load_image(mask_url).convert("RGB")init_image.save('init.png')mask_image.save('mask_image.png')prompt = "A majestic tiger sitting on a bench"image = pipeline(prompt, image=init_image, mask_image=mask_image, num_inference_steps=50, strength=0.80).images[0]image.save('temp.png')

結果

img2img: a portrait of a dog wearing a pearl earring
在這里插入圖片描述

inpainiting的結果：A majestic tiger sitting on a bench
在這里插入圖片描述

如果是隨機mask來inpaint的結果：（注意，黑色是要保留的）
在這里插入圖片描述

train a diffusion model

參考這個網站：
https://hf-mirror.com/docs/diffusers/tutorials/basic_training

注意，如果開頭的config里面push_to_hub設置為True的話，則需要改一下hub_model_id, 同時需要前面的

from huggingface_hub import notebook_login
notebook_login()

我試了一下有點問題，就直接設置為false了。

建議先把trianing跑完了，再仔細看trianing的代碼.

核心代碼：

for step, batch in enumerate(train_dataloader):clean_images = batch["images"]# Sample noise to add to the imagesnoise = torch.randn(clean_images.shape, device=clean_images.device)bs = clean_images.shape[0]# Sample a random timestep for each imagetimesteps = torch.randint(0, noise_scheduler.config.num_train_timesteps, (bs,), device=clean_images.device,dtype=torch.int64)# Add noise to the clean images according to the noise magnitude at each timestep# (this is the forward diffusion process)noisy_images = noise_scheduler.add_noise(clean_images, noise, timesteps)with accelerator.accumulate(model):# Predict the noise residualnoise_pred = model(noisy_images, timesteps, return_dict=False)[0]loss = F.mse_loss(noise_pred, noise)accelerator.backward(loss)accelerator.clip_grad_norm_(model.parameters(), 1.0)optimizer.step()lr_scheduler.step()optimizer.zero_grad()progress_bar.update(1)logs = {"loss": loss.detach().item(), "lr": lr_scheduler.get_last_lr()[0], "step": global_step}progress_bar.set_postfix(**logs)accelerator.log(logs, step=global_step)global_step += 1