@ -12,25 +12,28 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "1YwMUyt9LHG1"
},
"metadata": {},
"source": [
"# Disco Diffusion v5 - Now with 3D animation\n",
"\n",
"In case of confusion, Disco is the name of this notebook edit. The diffusion model in use is Katherine Crowson's fine-tuned 512x512 model\n",
"\n",
"For issues, join the [Disco Diffusion Discord](https://discord.gg/msEZBy4HxA) or message us on twitter at [@somnai_dreams](https://twitter.com/somnai_dreams) or [@gandamu](https://twitter.com/gandamu_ml)\n",
"\n",
"Credits & Changelog ⬇️
"For issues, join the [Disco Diffusion Discord](https://discord.gg/msEZBy4HxA) or message us on twitter at [@somnai_dreams](https://twitter.com/somnai_dreams) or [@gandamu](https://twitter.com/gandamu_ml)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wX5omb9C7Bjz"
},
"metadata": {},
"source": [
"### Credits & Changelog \u2b07\ufe0f"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Credits\n",
"\n",
"Original notebook by Katherine Crowson (https://github.com/crowsonkb, https://twitter.com/RiversHaveWings). It uses either OpenAI's 256x256 unconditional ImageNet or Katherine Crowson's fine-tuned 512x512 diffusion model (https://github.com/openai/guided-diffusion), together with CLIP (https://github.com/openai/CLIP) to connect text prompts with images.\n",
"\n",
"Modified by Daniel Russell (https://github.com/russelldc, https://twitter.com/danielrussruss) to include (hopefully) optimal params for quick generations in 15-100 timesteps rather than 1000, as well as more robust augmentations.\n",
@ -53,73 +56,101 @@
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "wDSYhyjqZQI9"
},
"outputs": [],
"cell_type": "markdown",
"metadata": {},
"source": [
"#### License"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"@title Licensed under the MIT License\n",
"\n",
"Copyright (c) 2021 Katherine Crowson \n",
"\n",
"Permission is hereby granted, free of charge, to any person obtaining a copy\n",
"of this software and associated documentation files (the \"Software\"), to deal\n",
"in the Software without restriction, including without limitation the rights\n",
"to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
"copies of the Software, and to permit persons to whom the Software is\n",
"furnished to do so, subject to the following conditions:\n",
"\n",
"The above copyright notice and this permission notice shall be included in\n",
"all copies or substantial portions of the Software.\n",
"\n",
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
"IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
"FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
"AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
"LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
"OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n",
"THE SOFTWARE.\n",
"\n",
"--\n",
"\n",
"MIT License\n",
"\n",
"Copyright (c) 2019 Intel ISL (Intel Intelligent Systems Lab)\n",
"\n",
"Permission is hereby granted, free of charge, to any person obtaining a copy\n",
"of this software and associated documentation files (the \"Software\"), to deal\n",
"in the Software without restriction, including without limitation the rights\n",
"to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
"copies of the Software, and to permit persons to whom the Software is\n",
"furnished to do so, subject to the following conditions:\n",
"\n",
"The above copyright notice and this permission notice shall be included in all\n",
"copies or substantial portions of the Software.\n",
"\n",
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
"IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
"FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
"AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
"LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
"OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
"SOFTWARE.\n",
"\n",
"--\n",
"\n",
"Licensed under the MIT License\n",
"\n",
"Copyright (c) 2021 Maxwell Ingham\n",
"\n",
"Copyright (c) 2022 Adam Letts \n",
"\n",
"Permission is hereby granted, free of charge, to any person obtaining a copy\n",
"of this software and associated documentation files (the \"Software\"), to deal\n",
"in the Software without restriction, including without limitation the rights\n",
"to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
"copies of the Software, and to permit persons to whom the Software is\n",
"furnished to do so, subject to the following conditions:\n",
"\n",
"The above copyright notice and this permission notice shall be included in\n",
"all copies or substantial portions of the Software.\n",
"\n",
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
"IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
"FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
"AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
"LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
"OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n",
"THE SOFTWARE."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# @title Licensed under the MIT License\n",
"\n",
"# Copyright (c) 2021 Katherine Crowson \n",
"\n",
"# Permission is hereby granted, free of charge, to any person obtaining a copy\n",
"# of this software and associated documentation files (the \"Software\"), to deal\n",
"# in the Software without restriction, including without limitation the rights\n",
"# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
"# copies of the Software, and to permit persons to whom the Software is\n",
"# furnished to do so, subject to the following conditions:\n",
"\n",
"# The above copyright notice and this permission notice shall be included in\n",
"# all copies or substantial portions of the Software.\n",
"\n",
"# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
"# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
"# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
"# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
"# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
"# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n",
"# THE SOFTWARE.\n",
"\n",
"# --\n",
"\n",
"# @title Licensed under the MIT License\n",
"\n",
"# Copyright (c) 2021 Maxwell Ingham \n",
"# Copyright (c) 2022 Adam Letts \n",
"\n",
"# Permission is hereby granted, free of charge, to any person obtaining a copy\n",
"# of this software and associated documentation files (the \"Software\"), to deal\n",
"# in the Software without restriction, including without limitation the rights\n",
"# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
"# copies of the Software, and to permit persons to whom the Software is\n",
"# furnished to do so, subject to the following conditions:\n",
"\n",
"# The above copyright notice and this permission notice shall be included in\n",
"# all copies or substantial portions of the Software.\n",
"\n",
"# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
"# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
"# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
"# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
"# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
"# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n",
"# THE SOFTWARE."
"#### Changelog"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "qFB3nwLSQI8X"
},
"outputs": [],
"metadata": {},
"source": [
"#@title <- View Changelog\n",
"\n",
"skip_for_run_all = True #@param {type: 'boolean'}\n",
"\n",
"if skip_for_run_all == False:\n",
@ -188,24 +219,27 @@
" v4.92 Update: Feb 20th 2022 - gandamu / Adam Letts\n",
"\n",
" Separated transform code\n",
"\n",
" v5.01 Update: Match 10th 2022 - gandamu / Adam Letts\n",
"\n",
" IPython magic commands replaced by Python code\n",
"\n",
" '''\n",
" )"
]
" )\n"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"metadata": {
"id": "XTu6AjLyFQUq"
},
"metadata": {},
"source": [
"#Tutorial"
"# Tutorial"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YR806W0wi3He"
},
"metadata": {},
"source": [
"**Diffusion settings (Defaults are heavily outdated)**\n",
"---\n",
@ -251,41 +285,56 @@
"`timestep_respacing` | Modify this value to decrease the number of timesteps. | ddim100\n",
"`diffusion_steps` || 1000\n",
"**Diffusion:**\n",
"`clip_models` | Models of CLIP to load. Typically the more, the better but they all come at a hefty VRAM cost. | ViT-B/32, ViT-B/16, RN50x4"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_9Eg9Kf5FlfK"
},
"source": [
"`clip_models` | Models of CLIP to load. Typically the more, the better but they all come at a hefty VRAM cost. | ViT-B/32, ViT-B/16, RN50x4\n",
"\n",
"# 1. Set Up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "qZ3rNuAWAewx"
},
"outputs": [],
"metadata": {},
"source": [
"#@title 1.1 Check GPU Status\n",
"!nvidia-smi -L"
]
"import subprocess\n",
"simple_nvidia_smi_display = False#@param {type:\"boolean\"}\n",
"if simple_nvidia_smi_display:\n",
" #!nvidia-smi\n",
" nvidiasmi_output = subprocess.run(['nvidia-smi', '-L'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" print(nvidiasmi_output)\n",
"else:\n",
" #!nvidia-smi -i 0 -e 0\n",
" nvidiasmi_output = subprocess.run(['nvidia-smi'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" print(nvidiasmi_output)\n",
" nvidiasmi_ecc_note = subprocess.run(['nvidia-smi', '-i', '0', '-e', '0'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" print(nvidiasmi_ecc_note)"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "yZsjzwS0YGo6"
},
"outputs": [],
"metadata": {},
"source": [
"#@title 1.2 Prepare Folders\n",
"import subprocess\n",
"import sys\n",
"import ipykernel\n",
"\n",
"def gitclone(url):\n",
" res = subprocess.run(['git', 'clone', url], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" print(res)\n",
"\n",
"def pipi(modulestr):\n",
" res = subprocess.run(['pip', 'install', modulestr], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" print(res)\n",
"\n",
"def pipie(modulestr):\n",
" res = subprocess.run(['git', 'install', '-e', modulestr], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" print(res)\n",
"\n",
"def wget(url, outputdir):\n",
" res = subprocess.run(['wget', url, '-P', f'{outputdir}'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" print(res)\n",
"\n",
"try:\n",
" from google.colab import drive\n",
@ -338,20 +387,18 @@
"\n",
"# libraries = f'{root_path}/libraries'\n",
"# createPath(libraries)"
]
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "JmbrcrhpBPC6"
},
"outputs": [],
"metadata": {},
"source": [
"#@title ### 1.3 Install and import dependencies\n",
"\n",
"from os.path import exists as path_exists\n",
"import pathlib, shutil\n",
"\n",
"if not is_colab:\n",
" # If running locally, there's a good chance your env will need this in order to not crash upon np.matmul() or similar operations.\n",
@ -373,23 +420,31 @@
"model_secondary_downloaded = False\n",
"\n",
"if is_colab:\n",
" !git clone https://github.com/openai/CLIP\n",
" # !git clone https://github.com/facebookresearch/SLIP.git\n",
" !git clone https://github.com/crowsonkb/guided-diffusion\n",
" !git clone https://github.com/assafshocher/ResizeRight.git\n",
" !pip install -e ./CLIP\n",
" !pip install -e ./guided-diffusion\n",
" !pip install lpips datetime timm\n",
" !apt install imagemagick\n",
" !git clone https://github.com/isl-org/MiDaS.git\n",
" !git clone https://github.com/alembics/disco-diffusion-dev.git\n",
" gitclone(\"https://github.com/openai/CLIP\")\n",
" #gitclone(\"https://github.com/facebookresearch/SLIP.git\")\n",
" gitclone(\"https://github.com/crowsonkb/guided-diffusion\")\n",
" gitclone(\"https://github.com/assafshocher/ResizeRight.git\")\n",
" pipie(\"./CLIP\")\n",
" pipie(\"./guided-diffusion\")\n",
" multipip_res = subprocess.run(['pip', 'install', 'lpips', 'datetime', 'timm', 'ftfy'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" print(multipip_res)\n",
" subprocess.run(['apt', 'install', 'imagemagick'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" gitclone(\"https://github.com/isl-org/MiDaS.git\")\n",
" gitclone(\"https://github.com/alembics/disco-diffusion.git\")\n",
" pipi(\"pytorch-lightning\")\n",
" pipi(\"omegaconf\")\n",
" pipi(\"einops\")\n",
" # Rename a file to avoid a name conflict..\n",
" !mv MiDaS/utils.py MiDaS/midas_utils.py\n",
" !cp disco-diffusion-dev/disco_xform_utils.py disco_xform_utils.py\n",
" try:\n",
" os.rename(\"MiDaS/utils.py\", \"MiDaS/midas_utils.py\")\n",
" shutil.copyfile(\"disco-diffusion/disco_xform_utils.py\", \"disco_xform_utils.py\")\n",
" except:\n",
" pass\n",
"\n",
"!mkdir model\n",
"if not path_exists(f'{model_path}'):\n",
" pathlib.Path(model_path).mkdir(parents=True, exist_ok=True)\n",
"if not path_exists(f'{model_path}/dpt_large-midas-2f21e586.pt'):\n",
" !wget https://github.com/intel-isl/DPT/releases/download/1_0/dpt_large-midas-2f21e586.pt -P {model_path} \n",
" wget(\"https://github.com/intel-isl/DPT/releases/download/1_0/dpt_large-midas-2f21e586.pt\", out=model_path) \n",
"\n",
"import sys\n",
"import torch\n",
@ -402,8 +457,9 @@
" torch.version.cuda.replace(\".\",\"\"),\n",
" f\"_pyt{pyt_version_str}\"\n",
" ])\n",
" !pip install fvcore iopath\n",
" !pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
" multipip_res = subprocess.run(['pip', 'install', 'fvcore', 'iopath'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" print(multipip_res)\n",
" subprocess.run(['pip', 'install', '--no-index', '--no-cache-dir', 'pytorch3d', '-f', f'https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
"\n",
"# sys.path.append('./SLIP')\n",
"sys.path.append('./ResizeRight')\n",
@ -443,10 +499,10 @@
"\n",
"#SuperRes\n",
"if is_colab:\n",
" !git clone https://github.com/CompVis/latent-diffusion.git \n",
" !git clone https://github.com/CompVis/taming-transformers \n",
" !pip install -e ./taming-transformers \n",
" !pip install ipywidgets omegaconf>=2.0.0 pytorch-lightning>=1.0.8 torch-fidelity einops wandb\n",
" gitclone(\"https://github.com/CompVis/latent-diffusion.git\") \n",
" gitclone(\"https://github.com/CompVis/taming-transformers\") \n",
" pipie(\"./taming-transformers\") \n",
" pipi(\" ipywidgets omegaconf>=2.0.0 pytorch-lightning>=1.0.8 torch-fidelity einops wandb\") \n",
"\n",
"#SuperRes\n",
"import ipywidgets as widgets\n",
@ -455,20 +511,22 @@
"sys.path.append('./taming-transformers')\n",
"from taming.models import vqgan # checking correct import from taming\n",
"from torchvision.datasets.utils import download_url\n",
"\n",
"if is_colab:\n",
" %cd '/content/latent-diffusion' \n",
" os.chdir('/content/latent-diffusion') \n",
"else:\n",
" %cd 'latent-diffusion'\n",
" #os.chdir('latent-diffusion')\n",
" sys.path.append('latent-diffusion')\n",
"from functools import partial\n",
"from ldm.util import instantiate_from_config\n",
"from ldm.modules.diffusionmodules.util import make_ddim_sampling_parameters, make_ddim_timesteps, noise_like\n",
"# from ldm.models.diffusion.ddim import DDIMSampler\n",
"from ldm.util import ismap\n",
"if is_colab:\n",
" %cd '/content' \n",
" os.chdir('/content') \n",
" from google.colab import files\n",
"else:\n",
" %cd $PROJECT_DIR \n",
" os.chdir(f'{PROJECT_DIR}') \n",
"from IPython.display import Image as ipyimg\n",
"from numpy import asarray\n",
"from einops import rearrange, repeat\n",
@ -481,11 +539,11 @@
"# AdaBins stuff\n",
"if USE_ADABINS:\n",
" if is_colab:\n",
" !git clone https://github.com/shariqfarooq123/AdaBins.git \n",
" gitclone(\"https://github.com/shariqfarooq123/AdaBins.git\") \n",
" if not path_exists(f'{model_path}/AdaBins_nyu.pt'):\n",
" !wget https://cloudflare-ipfs.com/ipfs/Qmd2mMnDLWePKmgfS8m6ntAg4nhV5VkUyAydYBp8cWWeB7/AdaBins_nyu.pt -P {model_path} \n",
" !mkdir pretrained \n",
" !cp -P {model_path}/AdaBins_nyu.pt pretrained/AdaBins_nyu.pt \n",
" wget(\"https://cloudflare-ipfs.com/ipfs/Qmd2mMnDLWePKmgfS8m6ntAg4nhV5VkUyAydYBp8cWWeB7/AdaBins_nyu.pt\", out=model_path) \n",
" pathlib.Path(\"pretrained\").mkdir(parents=True, exist_ok=True) \n",
" shutil.copyfile(f\"{model_path}/AdaBins_nyu.pt\", \"pretrained/AdaBins_nyu.pt\") \n",
" sys.path.append('./AdaBins')\n",
" from infer import InferenceHelper\n",
" MAX_ADABINS_AREA = 500000\n",
@ -498,16 +556,16 @@
"if torch.cuda.get_device_capability(DEVICE) == (8,0): ## A100 fix thanks to Emad\n",
" print('Disabling CUDNN for A100 gpu', file=sys.stderr)\n",
" torch.backends.cudnn.enabled = False"
]
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BLk3J0h3MtON"
},
"outputs": [],
"metadata": {},
"source": [
"#@title ### 1.4 Define Midas functions\n",
"\n",
"from midas.dpt_depth import DPTDepthModel\n",
"from midas.midas_net import MidasNet\n",
"from midas.midas_net_custom import MidasNet_small\n",
@ -606,19 +664,16 @@
" midas_model.to(DEVICE)\n",
"\n",
" print(f\"MiDaS '{midas_model_type}' depth model initialized.\")\n",
" return midas_model, midas_transform, net_w, net_h, resize_mode, normalization\n"
]
" return midas_model, midas_transform, net_w, net_h, resize_mode, normalization"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "FpZczxnOnPIU"
},
"outputs": [],
"metadata": {},
"source": [
"#@title 1.4 Define necessary functions\n",
"#@title 1.5 Define necessary functions\n",
"\n",
"# https://gist.github.com/adefossez/0646dbe9ed4005480a2407c62aac8869\n",
"\n",
@ -1365,20 +1420,16 @@
" }\n",
" # print('Settings:', setting_list)\n",
" with open(f\"{batchFolder}/{batch_name}({batchNum})_settings.txt\", \"w+\") as f: #save settings\n",
" json.dump(setting_list, f, ensure_ascii=False, indent=4)\n",
" "
]
" json.dump(setting_list, f, ensure_ascii=False, indent=4)"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "TI4oAu0N4ksZ"
},
"outputs": [],
"metadata": {},
"source": [
"#@title 1.5 Define the secondary diffusion model\n",
"#@title 1.6 Define the secondary diffusion model\n",
"\n",
"def append_dims(x, n):\n",
" return x[(Ellipsis, *(None,) * (n - x.ndim))]\n",
@ -1540,19 +1591,16 @@
" alphas, sigmas = map(partial(append_dims, n=v.ndim), t_to_alpha_sigma(t))\n",
" pred = input * alphas - v * sigmas\n",
" eps = input * sigmas + v * alphas\n",
" return DiffusionOutput(v, pred, eps)\n"
]
" return DiffusionOutput(v, pred, eps)"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "NJS2AUAnvn-D"
},
"outputs": [],
"metadata": {},
"source": [
"#@title 1.6 SuperRes Define\n",
"#@title 1.7 SuperRes Define\n",
"class DDIMSampler(object):\n",
" def __init__(self, model, schedule=\"linear\", **kwargs):\n",
" super().__init__()\n",
@ -2074,25 +2122,20 @@
" a.save(filepath)\n",
" return\n",
" print(f'Processing finished!')\n"
]
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"metadata": {
"id": "CQVtY1Ixnqx4"
},
"metadata": {},
"source": [
"# 2. Diffusion and CLIP model settings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "Fpbody2NCR7w"
},
"outputs": [],
"metadata": {},
"source": [
"#@markdown ####**Models Settings:**\n",
"diffusion_model = \"512x512_diffusion_uncond_finetune_008100\" #@param [\"256x256_diffusion_uncond\", \"512x512_diffusion_uncond_finetune_008100\"]\n",
@ -2139,12 +2182,12 @@
" model_256_downloaded = True\n",
" else: \n",
" print(\"256 Model SHA doesn't match, redownloading...\")\n",
" !wget --continue {model_256_link} -P {model_path} \n",
" wget(model_256_link, out=model_path) \n",
" model_256_downloaded = True\n",
" elif os.path.exists(model_256_path) and not check_model_SHA or model_256_downloaded == True:\n",
" print('256 Model already downloaded, check check_model_SHA if the file is corrupt')\n",
" else: \n",
" !wget --continue {model_256_link} -P {model_path} \n",
" wget(model_256_link, out=model_path) \n",
" model_256_downloaded = True\n",
"elif diffusion_model == '512x512_diffusion_uncond_finetune_008100':\n",
" if os.path.exists(model_512_path) and check_model_SHA:\n",
@ -2157,12 +2200,12 @@
" model_512_downloaded = True\n",
" else: \n",
" print(\"512 Model SHA doesn't match, redownloading...\")\n",
" !wget --continue {model_512_link} -P {model_path} \n",
" wget(model_512_link, out=model_path) \n",
" model_512_downloaded = True\n",
" elif os.path.exists(model_512_path) and not check_model_SHA or model_512_downloaded == True:\n",
" print('512 Model already downloaded, check check_model_SHA if the file is corrupt')\n",
" else: \n",
" !wget --continue {model_512_link} -P {model_path} \n",
" wget(model_512_link, out=model_path) \n",
" model_512_downloaded = True\n",
"\n",
"\n",
@ -2178,12 +2221,12 @@
" model_secondary_downloaded = True\n",
" else: \n",
" print(\"Secondary Model SHA doesn't match, redownloading...\")\n",
" !wget --continue {model_secondary_link} -P {model_path} \n",
" wget(model_secondary_link, out=model_path) \n",
" model_secondary_downloaded = True\n",
" elif os.path.exists(model_secondary_path) and not check_model_SHA or model_secondary_downloaded == True:\n",
" print('Secondary Model already downloaded, check check_model_SHA if the file is corrupt')\n",
" else: \n",
" !wget --continue {model_secondary_link} -P {model_path} \n",
" wget(model_secondary_link, out=model_path) \n",
" model_secondary_downloaded = True\n",
"\n",
"model_config = model_and_diffusion_defaults()\n",
@ -2247,7 +2290,7 @@
"if SLIPB16:\n",
" SLIPB16model = SLIP_VITB16(ssl_mlp_dim=4096, ssl_emb_dim=256)\n",
" if not os.path.exists(f'{model_path}/slip_base_100ep.pt'):\n",
" !wget https://dl.fbaipublicfiles.com/slip/slip_base_100ep.pt -P {model_path} \n",
" wget(\"https://dl.fbaipublicfiles.com/slip/slip_base_100ep.pt\", out=model_path) \n",
" sd = torch.load(f'{model_path}/slip_base_100ep.pt')\n",
" real_sd = {}\n",
" for k, v in sd['state_dict'].items():\n",
@ -2261,7 +2304,7 @@
"if SLIPL16:\n",
" SLIPL16model = SLIP_VITL16(ssl_mlp_dim=4096, ssl_emb_dim=256)\n",
" if not os.path.exists(f'{model_path}/slip_large_100ep.pt'):\n",
" !wget https://dl.fbaipublicfiles.com/slip/slip_large_100ep.pt -P {model_path} \n",
" wget(\"https://dl.fbaipublicfiles.com/slip/slip_large_100ep.pt\", out=model_path) \n",
" sd = torch.load(f'{model_path}/slip_large_100ep.pt')\n",
" real_sd = {}\n",
" for k, v in sd['state_dict'].items():\n",
@ -2273,26 +2316,21 @@
" clip_models.append(SLIPL16model)\n",
"\n",
"normalize = T.Normalize(mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])\n",
"lpips_model = lpips.LPIPS(net='vgg').to(device)"
]
"lpips_model = lpips.LPIPS(net='vgg').to(device)\n"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"metadata": {
"id": "kjtsXaszn-bB"
},
"metadata": {},
"source": [
"# 3. Settings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "U0PwzFZbLfcy"
},
"outputs": [],
"metadata": {},
"source": [
"#@markdown ####**Basic Settings:**\n",
"batch_name = 'TimeToDisco' #@param{type: 'string'}\n",
@ -2330,25 +2368,20 @@
"#Make folder for batch\n",
"batchFolder = f'{outDirPath}/{batch_name}'\n",
"createPath(batchFolder)\n"
]
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"metadata": {
"id": "CnkTNXJAPzL2"
},
"metadata": {},
"source": [
"###Animation Settings"
"### Animation Settings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "djPY2_4kHgV2"
},
"outputs": [],
"metadata": {},
"source": [
"#@markdown ####**Animation Mode:**\n",
"animation_mode = 'None' #@param ['None', '2D', '3D', 'Video Input'] {type:'string'}\n",
@ -2372,11 +2405,13 @@
" createPath(videoFramesFolder)\n",
" print(f\"Exporting Video Frames (1 every {extract_nth_frame})...\")\n",
" try:\n",
" !rm {videoFramesFolder}/*.jpg\n",
" for f in pathlib.Path(f'{videoFramesFolder}').glob('*.jpg'):\n",
" f.unlink()\n",
" except:\n",
" print('')\n",
" vf = f'\"select=not(mod(n\\,{extract_nth_frame}))\"'\n",
" !ffmpeg -i {video_init_path} -vf {vf} -vsync vfr -q:v 2 -loglevel error -stats {videoFramesFolder}/%04d.jpg\n",
" subprocess.run(['ffmpeg', '-i', f'{video_init_path}', '-vf', f'{vf}', '-vsync', 'vfr', '-q:v', '2', '-loglevel', 'error', '-stats', f'{videoFramesFolder}/%04d.jpg'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
" #!ffmpeg -i {video_init_path} -vf {vf} -vsync vfr -q:v 2 -loglevel error -stats {videoFramesFolder}/%04d.jpg\n",
"\n",
"\n",
"#@markdown ---\n",
@ -2654,14 +2689,14 @@
" translation_z = float(translation_z)\n",
" rotation_3d_x = float(rotation_3d_x)\n",
" rotation_3d_y = float(rotation_3d_y)\n",
" rotation_3d_z = float(rotation_3d_z)"
]
" rotation_3d_z = float(rotation_3d_z)\n"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"metadata": {
"id": "u1VHzHvNx5fd"
},
"metadata": {},
"source": [
"### Extra Settings\n",
" Partial Saves, Diffusion Sharpening, Advanced Settings, Cutn Scheduling"
@ -2669,12 +2704,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "lCLMxtILyAHA"
},
"outputs": [],
"metadata": {},
"source": [
"#@markdown ####**Saving:**\n",
"\n",
@ -2745,27 +2775,22 @@
"cut_overview = \"[12]*400+[4]*600\" #@param {type: 'string'} \n",
"cut_innercut =\"[4]*400+[12]*600\"#@param {type: 'string'} \n",
"cut_ic_pow = 1#@param {type: 'number'} \n",
"cut_icgray_p = \"[0.2]*400+[0]*600\"#@param {type: 'string'} \n",
"\n"
]
"cut_icgray_p = \"[0.2]*400+[0]*600\"#@param {type: 'string'}\n"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"metadata": {
"id": "XIwh5RvNpk4K"
},
"metadata": {},
"source": [
"###Prompts\n",
"### Prompts\n",
"`animation_mode: None` will only use the first set. `animation_mode: 2D / Video` will run through them per the set frames and hold on the last one."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BGBzhk3dpcGO"
},
"outputs": [],
"metadata": {},
"source": [
"text_prompts = {\n",
" 0: [\"A beautiful painting of a singular lighthouse, shining its light across a tumultuous sea of blood by greg rutkowski and thomas kinkade, Trending on artstation.\", \"yellow color scheme\"],\n",
@ -2774,26 +2799,21 @@
"\n",
"image_prompts = {\n",
" # 0:['ImagePromptsWorkButArentVeryGood.png:2',],\n",
"}"
]
"}\n"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"metadata": {
"id": "Nf9hTc8YLoLx"
},
"metadata": {},
"source": [
"# 4. Diffuse!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "LHLiO56OfwgD"
},
"outputs": [],
"metadata": {},
"source": [
"#@title Do the Run!\n",
"#@markdown `n_batches` ignored with animation modes.\n",
@ -2961,26 +2981,21 @@
"finally:\n",
" print('Seed used:', seed)\n",
" gc.collect()\n",
" torch.cuda.empty_cache()"
]
" torch.cuda.empty_cache()\n"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"metadata": {
"id": "EZUg3bfzazgW"
},
"metadata": {},
"source": [
"# 5. Create the video"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "HV54fuU3pMzJ"
},
"outputs": [],
"metadata": {},
"source": [
"# @title ### **Create video**\n",
"#@markdown Video file will save in the same folder as your images.\n",
@ -3056,27 +3071,16 @@
" # mp4 = open(filepath,'rb').read()\n",
" # data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
" # display.HTML(f'<video width=400 controls><source src=\"{data_url}\" type=\"video/mp4\"></video>')"
]
],
"outputs": [],
"execution_count": null
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [
"1YwMUyt9LHG1",
"XTu6AjLyFQUq",
"_9Eg9Kf5FlfK",
"CnkTNXJAPzL2",
"u1VHzHvNx5fd"
],
"machine_shape": "hm",
"name": "Disco Diffusion v4.1 [w/ Video Inits, Recovery & DDIM Sharpen].ipynb",
"private_outputs": true,
"provenance": [],
"include_colab_link": true
},
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
@ -3089,9 +3093,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7 "
"version": "3.6.1 "
}
},
"nbformat": 4,
"nbformat_minor": 0
"nbformat_minor": 4
}
aletts
3 years ago