{
"cells": [
{
"cell_type": "markdown",
"source": [
"# Paperspace Stable Diffusion Notebook\n",
"\n",
"This Notebook contains instructions and easy steps for generating images with Stable Diffusion on any of Paperspace's GPUs with little set up required. Follow the instructions in the markdown cells in this Notebook to quickly launch the Stable Diffusion Web UI or begin generating images using the Diffusers Stable Diffusion Pipeline.\n",
"\n",
"Contents:\n",
"\n",
"- Use the Stable Diffusion Web UI\n",
" - Loading in the Stable Diffusion models for the Web UI with support for v1-5 and v2 models\n",
"- The Diffusers Stable Diffusion Pipeline\n",
" - Loading in the Stable Diffusion models for the Diffusers StableDiffusionPipeline with support for v1-4, v1-5 and v2 models\n",
" - Access additional models: log in to HuggingFace for online access model files\n",
" - Inference with the Diffusers pipeline\n",
"- CLIP reranking\n",
"- GFPGAN face restoration and image upscaling\n",
"- CodeFormer face restoration and image upscaling"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Setup\n",
"\n",
"First, run the cell below to get everything we need for this Notebook to run."
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"!pip install --upgrade git+https://github.com/huggingface/diffusers.git \n",
"!pip install -U transformers accelerate scipy\n",
"!pip install flax==0.5.0 --no-deps\n",
"!pip install msgpack rich optax accelerate ftfy scipy\n",
"!pip install ipywidgets\n",
"!jupyter nbextension enable --py widgetsnbextension\n",
"!git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui\n",
"!mkdir outputs\n",
"!pip uninstall -y torch torchvision torchaudio\n",
"!pip3 install torch torchvision torchaudio --no-cache-dir --index-url https://download.pytorch.org/whl/cu118\n"
],
"outputs": [],
"metadata": {
"collapsed": true,
"execution": {
"iopub.execute_input": "2022-11-29T01:17:34.402699Z",
"iopub.status.busy": "2022-11-29T01:17:34.401879Z",
"iopub.status.idle": "2022-11-29T01:17:58.874403Z",
"shell.execute_reply": "2022-11-29T01:17:58.873489Z",
"shell.execute_reply.started": "2022-11-29T01:17:34.402612Z"
},
"jupyter": {
"outputs_hidden": true
}
}
},
{
"cell_type": "markdown",
"source": [
"## Use the Stable Diffusion Web UI\n",
"\n",
"Run the cell below to setup, and launch the Automatic1111 Stable Diffusion Web UI from this notebook. Click the Gradio link after the setup completes to access the Web UI in your local browser. \n",
"\n",
"Skip this step to use the Diffusers library methodology.\n",
"\n",
"### Loading in the Stable Diffusion models for the Web UI \n",
"\n",
"To make accessing the Stable Diffusion models easy and not take up any storage, we have added the Stable Diffusion v1-5 and v2 models as mountable public datasets!\n",
"\n",
"> These files will not count towards the storage limit for free notebooks.\n",
"\n",
"To access the model files:\n",
"\n",
"First, navigate to the \"Data Sources\" tab using the navigator on the far left of the page.\n",
"\n",
"Next, click \"Public\" to switch into the Gradient Public Datasets, and scroll down until you find the Stable Diffusion datasets near the bottom of the list. You can mount `stable-diffusion-classic` for access to the v1-5 model checkpoints for the web UI, `stable-diffusion-diffusers` for the v1-5 models as Diffusers StableDiffusionPipeline binaries, `stable-diffusion-classic-v2` for access to the v2 model checkpoints and config file for use with the web UI, and `stable-diffusion-diffusers-v2` for access to the v2 models as Diffusers StableDiffusionPipeline binaries.\n",
"\n",
"Finally, click \"mount\" to make any of these Public Datasets accessible from the \"datasets\" directory. This directory is in the root folder, so access it with the path `~/../datasets/stable-diffusion-classic/SDv1.5.ckpt`. For example, after mounting, the v1-5 checkpoint can be found at `~/../datasets/stable-diffusion-classic/SDv1.5.ckpt`\n",
"\n",
"See the video below for a video guide to for help mounting these files. \n"
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# Launch the Stable Diffusion Web UI. \n",
"# Be sure to hash out whichever you are not using.\n",
"%cd stable-diffusion-webui\n",
"\n",
"## Launch Web UI for Stable Diffusion v2 models. Note: this prevents users from using checkpoints and embeddings trained on v1-X models.\n",
"!python launch.py --share --config ~/../datasets/stable-diffusion-classic-v2/768-v-ema.yaml --ckpt ~/../datasets/stable-diffusion-classic-v2/768-v-ema.ckpt\n",
"\n",
"## Launch Web UI for Stable Diffusion v1-x models.\n",
"!python launch.py --share --ckpt ../../datasets/stable-diffusion-classic/SDv1.5.ckpt\n"
],
"outputs": [],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"# The Diffusers Stable Diffusion Pipeline\n",
"\n",
"## Loading in the Stable Diffusion v1-5 and v2 models for the Diffusers StableDiffusionPipeline\n",
"\n",
"To make accessing the Stable Diffusion models easy and not take up any storage, we have added the Stable Diffusion models as mountable public datasets! To access the models this way, simply navigate to the \"Data Sources\" tab using the navigator on the far left of the page. \n",
"\n",
"Then click \"Public\" to switch into the Gradient Public Datasets, and scroll down until you find the datasets starting with \"stable-diffusion\" near the bottom of the list. We recommend mounting each of these public datasets: `stable-diffusion-classic` (v1-5 checkpoint), `stable-diffusion-diffusers` (v1-5 diffusers), `stable-diffusion-classic-v2` (v2 checkpoint & config), and `stable-diffusion-diffusers-v2` (v2 diffusers). \n",
"\n",
"Click \"mount\" to make each of them accessible from the \"datasets\" directory. This directory is in the root folder, so access the two directories with the paths `~/../datasets/` and `../datasets/stable-diffusion-diffusers-v2/stable-diffusion-2/`\n",
"\n",
"\n"
],
"metadata": {}
},
{
"cell_type": "markdown",
"source": [
"## Access additional models: log in to HuggingFace for online access to v1-4 and v2 model files\n",
"\n",
"In order to access the models from CompVis for Stable Diffusion, you must follow three steps:\n",
"\n",
"1. You must acknowledge and agree to their user requirements and license for their models. You can do so by reading the instructions found on these pages: \n",
"- [CompVis/stable-diffusion-v1-4](https://huggingface.co/CompVis/stable-diffusion-v1-4)\n",
"- [RunwayML/stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5)\n",
"- [StabilityAI/stable-diffusion-2](https://huggingface.co/stabilityai/stable-diffusion-2) \n",
"\n",
"2. You must login to Huggingface, and then create and retrieve an access token (found here: https://huggingface.co/settings/tokens)\n",
"\n",
"3. Finally, replace the segment of the cell below `` with your own token, and run the cell. \n",
"\n",
"If you follow these steps, you will be able to access the model for free!\n",
"\n"
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": 2,
"source": [
"!python login.py --token "
],
"outputs": [],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T01:18:21.947625Z",
"iopub.status.busy": "2022-11-29T01:18:21.946869Z",
"iopub.status.idle": "2022-11-29T01:18:22.589006Z",
"shell.execute_reply": "2022-11-29T01:18:22.588022Z",
"shell.execute_reply.started": "2022-11-29T01:18:21.947594Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"## Inference with the Diffusers pipeline\n",
"\n",
"Use the following cells to begin synthesizing images. Use the radio buttons in the cell below to select which model type to use. \n",
"\n",
"### Setup\n",
"\n",
"Use the following cell to set up your inference by\n",
"\n",
"- Choose which Stable Diffusion checkpoint to use\n",
"- Creating the outputs directory\n",
"- Choose whether to use FP16 or FP32 (use FP16 for Free-GPU Notebooks)\n",
"- import packages"
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"import torch\n",
"from torch import autocast\n",
"from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler\n",
"import ipywidgets as widgets\n",
"from ipywidgets import interact\n",
"\n",
"%cd ~/../notebooks\n",
"\n",
"## Use the lists below to select our model type, image output resolution, and the computer number format for inference\n",
"device = 'cuda'\n",
"## We recommend either v1-5 (index 1) or v2 (index 2)\n",
"model_id = ['CompVis/stable-diffusion-v1-4',\"../datasets/stable-diffusion-diffusers/stable-diffusion-v1-5/\",'../datasets/stable-diffusion-diffusers-v2/stable-diffusion-2/']\n",
"## We strongly recommend using the v1-5 and v1-4 models with size 512, and the v2 models with size 768.\n",
"size = [512, 768]\n",
"## If you are using a GPU with ~ 8 GBs of RAM, like the Free-GPU Notebooks, then use torch.float16\n",
"## otherwise, either selection works\n",
"dtype = [torch.float16, torch.float32]\n",
"\n",
"\n",
"# Set your values here - defaults are for v2, size 768 x 768 with FP16 computer number format\n",
"model_ = model_id[2]\n",
"size_ = size[1] \n",
"precision = dtype[0]"
],
"outputs": [],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T02:07:24.666530Z",
"iopub.status.busy": "2022-11-29T02:07:24.665580Z",
"iopub.status.idle": "2022-11-29T02:07:24.687341Z",
"shell.execute_reply": "2022-11-29T02:07:24.686545Z",
"shell.execute_reply.started": "2022-11-29T02:07:24.666484Z"
}
}
},
{
"cell_type": "code",
"execution_count": 28,
"source": [
"# Depending on the selections made above, the type of model loaded will change using this short if/else statement\n",
"\n",
"print(f'Now loading Stable Diffusion {model_} model ({precision})...')\n",
"scheduler = EulerDiscreteScheduler.from_pretrained('stabilityai/stable-diffusion-2', subfolder=\"scheduler\")\n",
"if precision == torch.float16:\n",
" if model_ == '../datasets/stable-diffusion-diffusers-v2/stable-diffusion-2/':\n",
" pipe = StableDiffusionPipeline.from_pretrained(model_, torch_dtype=torch.float16, revision=\"fp16\", scheduler = scheduler)\n",
" pipe = pipe.to(device)\n",
" else:\n",
" pipe = StableDiffusionPipeline.from_pretrained(model_, torch_dtype=torch.float16, revision=\"fp16\")\n",
" pipe = pipe.to(device)\n",
"else:\n",
" if model_ == '../datasets/stable-diffusion-diffusers-v2/stable-diffusion-2/':\n",
" pipe = StableDiffusionPipeline.from_pretrained(model_, scheduler = scheduler)\n",
" pipe = pipe.to(device)\n",
" else:\n",
" pipe = StableDiffusionPipeline.from_pretrained(model_)\n",
" pipe = pipe.to(device)\n",
"print('Model now loaded. Head to next cell to generate images.')"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Now loading Stable Diffusion v1-5 model (FP32)...\n",
"Model now loaded. Head to next cell to generate images.\n"
]
}
],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T02:05:59.397543Z",
"iopub.status.busy": "2022-11-29T02:05:59.396509Z",
"iopub.status.idle": "2022-11-29T02:06:04.176412Z",
"shell.execute_reply": "2022-11-29T02:06:04.175519Z",
"shell.execute_reply.started": "2022-11-29T02:05:59.397490Z"
}
}
},
{
"cell_type": "code",
"execution_count": 29,
"source": [
"sample_num = 5\n",
"lst = []\n",
"\n",
"print(f'Generating samples from Stable Diffusion {model_} checkpoint ({precision})')\n",
"\n",
"prompt = 'a corgi astronaut on mars'\n",
"for j in range(sample_num):\n",
" a = pipe(prompt,\n",
" guidance_scale=7.5,\n",
" height = size_,\n",
" width = size_,\n",
" num_inference_steps=50)['images'][0]\n",
" lst.append(a)\n",
" display(a)\n",
" a.save(f'outputs/gen-image-{j}.png')"
],
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Generating samples from Stable Diffusion v1-5 checkpoint (FP32)\n"
]
},
{
"output_type": "display_data",
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "87e69479839747e6af6d2102447d54e8",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/50 [00:00"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "cbd5b4988cea4684bc8a1db09dda356a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/50 [00:00"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "79afb075633b47e5a8106c6ddf32c7b9",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/50 [00:00"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a979079b0e594d2ea97a3b292f7df7eb",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/50 [00:00"
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d9dfc3a37b654590bea0dd2d119c8333",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/50 [00:00"
]
},
"metadata": {}
}
],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T02:06:04.178009Z",
"iopub.status.busy": "2022-11-29T02:06:04.177770Z",
"iopub.status.idle": "2022-11-29T02:06:31.852489Z",
"shell.execute_reply": "2022-11-29T02:06:31.851503Z",
"shell.execute_reply.started": "2022-11-29T02:06:04.177987Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"# CLIP reranking\n",
"\n",
"Find the best of your images from the generated samples by ranking them with CLIP by their accuracy to the prompt. \n",
"\n",
"Code adapted from Boris Dayma's [DALL-E Mini Inference Pipeline](https://github.com/borisdayma/dalle-mini/blob/main/tools/inference/inference_pipeline.ipynb) "
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"from transformers import CLIPProcessor, FlaxCLIPModel\n",
"import jax\n",
"import jax.numpy as jnp\n",
"from flax.jax_utils import replicate\n",
"from functools import partial\n",
"\n",
"\n",
"# CLIP model\n",
"CLIP_REPO = \"openai/clip-vit-base-patch32\"\n",
"CLIP_COMMIT_ID = None\n",
"\n",
"# Load CLIP\n",
"clip, clip_params = FlaxCLIPModel.from_pretrained(\n",
" CLIP_REPO, revision=CLIP_COMMIT_ID, dtype=jnp.float16, _do_init=False\n",
")\n",
"clip_processor = CLIPProcessor.from_pretrained(CLIP_REPO, revision=CLIP_COMMIT_ID)\n",
"clip_params = replicate(clip_params)\n",
"\n",
"# score images\n",
"@partial(jax.pmap, axis_name=\"batch\")\n",
"def p_clip(inputs, params):\n",
" logits = clip(params=params, **inputs).logits_per_image\n",
" return logits"
],
"outputs": [],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T01:22:10.381720Z",
"iopub.status.busy": "2022-11-29T01:22:10.381549Z",
"iopub.status.idle": "2022-11-29T01:22:33.498295Z",
"shell.execute_reply": "2022-11-29T01:22:33.497675Z",
"shell.execute_reply.started": "2022-11-29T01:22:10.381702Z"
}
}
},
{
"cell_type": "code",
"execution_count": 6,
"source": [
"from flax.training.common_utils import shard\n",
"import numpy as np\n",
"# get clip scores\n",
"clip_inputs = clip_processor(\n",
" text=prompt * jax.device_count(),\n",
" images=lst,\n",
" return_tensors=\"np\",\n",
" padding=\"max_length\",\n",
" max_length=77,\n",
" truncation=True,\n",
").data\n",
"logits = p_clip(shard(clip_inputs), clip_params)\n",
"\n",
"out = list(reversed(sorted(zip(logits[0], lst))))"
],
"outputs": [],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T01:22:33.501791Z",
"iopub.status.busy": "2022-11-29T01:22:33.501157Z",
"iopub.status.idle": "2022-11-29T01:22:38.836102Z",
"shell.execute_reply": "2022-11-29T01:22:38.835334Z",
"shell.execute_reply.started": "2022-11-29T01:22:33.501767Z"
}
}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"for idx, v in enumerate(out):\n",
" display(v[1])\n",
" print(f\"Score: {v[0][0]:.2f}\\n\")\n",
" \n"
],
"outputs": [],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T01:22:38.839633Z",
"iopub.status.busy": "2022-11-29T01:22:38.839457Z",
"iopub.status.idle": "2022-11-29T01:22:40.073647Z",
"shell.execute_reply": "2022-11-29T01:22:40.072812Z",
"shell.execute_reply.started": "2022-11-29T01:22:38.839614Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"# GFPGAN\n",
"\n",
"GFPGAN aims at developing a Practical Algorithm for Real-world Face Restoration. It leverages rich and diverse priors encapsulated in a pretrained face GAN (e.g., StyleGAN2) for blind face restoration. GFPGAN is combined with a suite of other restoration and upscaling models to help make your pictures much clearer and cleaner. \n",
"\n",
"credit to: https://github.com/TencentARC/GFPGAN"
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"!git clone https://github.com/TencentARC/GFPGAN.git\n",
"\n",
"%cd GFPGAN\n",
"# Install basicsr - https://github.com/xinntao/BasicSR\n",
"# We use BasicSR for both training and inference\n",
"# Install facexlib - https://github.com/xinntao/facexlib\n",
"# We use face detection and face restoration helper in the facexlib package\n",
"!pip install facexlib\n",
"\n",
"!pip install - r requirements.txt\n",
"!python setup.py develop\n",
"!pip install realesrgan basicsr optax\n",
"\n",
"# If you want to enhance the background (non-face) regions with Real-ESRGAN,\n",
"# you also need to install the realesrgan package\n",
"!pip install realesrgan\n",
"\n",
"# get model\n",
"%cd GFPGAN/experiments/pretrained_models\n",
"!wget https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.3.pth\n",
"%cd ~/../notebooks"
],
"outputs": [],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T01:22:40.077068Z",
"iopub.status.busy": "2022-11-29T01:22:40.076786Z",
"iopub.status.idle": "2022-11-29T01:24:30.255035Z",
"shell.execute_reply": "2022-11-29T01:24:30.254322Z",
"shell.execute_reply.started": "2022-11-29T01:22:40.077042Z"
}
}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"%cd GFPGAN\n",
"!python inference_gfpgan.py -i ../outputs -o results -v 1.3 -s 2\n",
"%cd ~/../notebooks\n"
],
"outputs": [],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T01:24:30.258351Z",
"iopub.status.busy": "2022-11-29T01:24:30.258165Z",
"iopub.status.idle": "2022-11-29T01:24:57.225130Z",
"shell.execute_reply": "2022-11-29T01:24:57.224130Z",
"shell.execute_reply.started": "2022-11-29T01:24:30.258328Z"
}
}
},
{
"cell_type": "markdown",
"source": [
"# CodeFormer\n",
"\n",
"'Transformer-based prediction network, named CodeFormer, to model global composition and context of the low-quality faces for code prediction, enabling the discovery of natural faces that closely approximate the target faces even when the inputs are severely degraded.' - Project credit to [Source](https://shangchenzhou.com/projects/CodeFormer/)\n",
"\n",
"This alternative to GFPGAN also does a great job of upscaling images. Much like above, you can run the next two cells to upscale the full images from our outputs directory.\n"
],
"metadata": {}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"# Install and setup\n",
"# git clone this repository\n",
"!git clone https://github.com/sczhou/CodeFormer\n",
"%cd CodeFormer\n",
"# install python dependencies\n",
"!pip install realesrgan basicsr optax lpips\n",
"!python scripts/download_pretrained_models.py facelib\n",
"!python scripts/download_pretrained_models.py CodeFormer\n",
"%cd .."
],
"outputs": [],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T01:25:18.141158Z",
"iopub.status.busy": "2022-11-29T01:25:18.140788Z",
"iopub.status.idle": "2022-11-29T01:25:21.199350Z",
"shell.execute_reply": "2022-11-29T01:25:21.198315Z",
"shell.execute_reply.started": "2022-11-29T01:25:18.141128Z"
}
}
},
{
"cell_type": "code",
"execution_count": null,
"source": [
"%cd CodeFormer\n",
"!python inference_codeformer.py -w 0.7 --input_path ../outputs\n",
"%cd .."
],
"outputs": [],
"metadata": {
"execution": {
"iopub.execute_input": "2022-11-29T01:25:23.778593Z",
"iopub.status.busy": "2022-11-29T01:25:23.778254Z",
"iopub.status.idle": "2022-11-29T01:25:38.477021Z",
"shell.execute_reply": "2022-11-29T01:25:38.476181Z",
"shell.execute_reply.started": "2022-11-29T01:25:23.778566Z"
}
}
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}