diff --git a/Disco_Diffusion.ipynb b/Disco_Diffusion.ipynb
index c2df85e..b00db90 100644
--- a/Disco_Diffusion.ipynb
+++ b/Disco_Diffusion.ipynb
@@ -12,25 +12,28 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "1YwMUyt9LHG1"
- },
+ "metadata": {},
"source": [
"# Disco Diffusion v5 - Now with 3D animation\n",
"\n",
"In case of confusion, Disco is the name of this notebook edit. The diffusion model in use is Katherine Crowson's fine-tuned 512x512 model\n",
"\n",
- "For issues, join the [Disco Diffusion Discord](https://discord.gg/msEZBy4HxA) or message us on twitter at [@somnai_dreams](https://twitter.com/somnai_dreams) or [@gandamu](https://twitter.com/gandamu_ml)\n",
- "\n",
- "Credits & Changelog ⬇️\n"
+ "For issues, join the [Disco Diffusion Discord](https://discord.gg/msEZBy4HxA) or message us on twitter at [@somnai_dreams](https://twitter.com/somnai_dreams) or [@gandamu](https://twitter.com/gandamu_ml)"
]
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "wX5omb9C7Bjz"
- },
+ "metadata": {},
+ "source": [
+ "### Credits & Changelog \u2b07\ufe0f"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
"source": [
+ "#### Credits\n",
+ "\n",
"Original notebook by Katherine Crowson (https://github.com/crowsonkb, https://twitter.com/RiversHaveWings). It uses either OpenAI's 256x256 unconditional ImageNet or Katherine Crowson's fine-tuned 512x512 diffusion model (https://github.com/openai/guided-diffusion), together with CLIP (https://github.com/openai/CLIP) to connect text prompts with images.\n",
"\n",
"Modified by Daniel Russell (https://github.com/russelldc, https://twitter.com/danielrussruss) to include (hopefully) optimal params for quick generations in 15-100 timesteps rather than 1000, as well as more robust augmentations.\n",
@@ -53,73 +56,101 @@
]
},
{
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "wDSYhyjqZQI9"
- },
- "outputs": [],
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### License"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "@title Licensed under the MIT License\n",
+ "\n",
+ "Copyright (c) 2021 Katherine Crowson \n",
+ "\n",
+ "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
+ "of this software and associated documentation files (the \"Software\"), to deal\n",
+ "in the Software without restriction, including without limitation the rights\n",
+ "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
+ "copies of the Software, and to permit persons to whom the Software is\n",
+ "furnished to do so, subject to the following conditions:\n",
+ "\n",
+ "The above copyright notice and this permission notice shall be included in\n",
+ "all copies or substantial portions of the Software.\n",
+ "\n",
+ "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
+ "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
+ "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
+ "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
+ "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
+ "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n",
+ "THE SOFTWARE.\n",
+ "\n",
+ "--\n",
+ "\n",
+ "MIT License\n",
+ "\n",
+ "Copyright (c) 2019 Intel ISL (Intel Intelligent Systems Lab)\n",
+ "\n",
+ "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
+ "of this software and associated documentation files (the \"Software\"), to deal\n",
+ "in the Software without restriction, including without limitation the rights\n",
+ "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
+ "copies of the Software, and to permit persons to whom the Software is\n",
+ "furnished to do so, subject to the following conditions:\n",
+ "\n",
+ "The above copyright notice and this permission notice shall be included in all\n",
+ "copies or substantial portions of the Software.\n",
+ "\n",
+ "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
+ "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
+ "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
+ "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
+ "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
+ "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
+ "SOFTWARE.\n",
+ "\n",
+ "--\n",
+ "\n",
+ "Licensed under the MIT License\n",
+ "\n",
+ "Copyright (c) 2021 Maxwell Ingham\n",
+ "\n",
+ "Copyright (c) 2022 Adam Letts \n",
+ "\n",
+ "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
+ "of this software and associated documentation files (the \"Software\"), to deal\n",
+ "in the Software without restriction, including without limitation the rights\n",
+ "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
+ "copies of the Software, and to permit persons to whom the Software is\n",
+ "furnished to do so, subject to the following conditions:\n",
+ "\n",
+ "The above copyright notice and this permission notice shall be included in\n",
+ "all copies or substantial portions of the Software.\n",
+ "\n",
+ "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
+ "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
+ "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
+ "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
+ "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
+ "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n",
+ "THE SOFTWARE."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
"source": [
- "# @title Licensed under the MIT License\n",
- "\n",
- "# Copyright (c) 2021 Katherine Crowson \n",
- "\n",
- "# Permission is hereby granted, free of charge, to any person obtaining a copy\n",
- "# of this software and associated documentation files (the \"Software\"), to deal\n",
- "# in the Software without restriction, including without limitation the rights\n",
- "# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
- "# copies of the Software, and to permit persons to whom the Software is\n",
- "# furnished to do so, subject to the following conditions:\n",
- "\n",
- "# The above copyright notice and this permission notice shall be included in\n",
- "# all copies or substantial portions of the Software.\n",
- "\n",
- "# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
- "# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
- "# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
- "# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
- "# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
- "# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n",
- "# THE SOFTWARE.\n",
- "\n",
- "# --\n",
- "\n",
- "# @title Licensed under the MIT License\n",
- "\n",
- "# Copyright (c) 2021 Maxwell Ingham \n",
- "# Copyright (c) 2022 Adam Letts \n",
- "\n",
- "# Permission is hereby granted, free of charge, to any person obtaining a copy\n",
- "# of this software and associated documentation files (the \"Software\"), to deal\n",
- "# in the Software without restriction, including without limitation the rights\n",
- "# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
- "# copies of the Software, and to permit persons to whom the Software is\n",
- "# furnished to do so, subject to the following conditions:\n",
- "\n",
- "# The above copyright notice and this permission notice shall be included in\n",
- "# all copies or substantial portions of the Software.\n",
- "\n",
- "# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
- "# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
- "# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
- "# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
- "# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
- "# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n",
- "# THE SOFTWARE."
+ "#### Changelog"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "qFB3nwLSQI8X"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"#@title <- View Changelog\n",
- "\n",
"skip_for_run_all = True #@param {type: 'boolean'}\n",
"\n",
"if skip_for_run_all == False:\n",
@@ -188,24 +219,27 @@
" v4.92 Update: Feb 20th 2022 - gandamu / Adam Letts\n",
"\n",
" Separated transform code\n",
+ "\n",
+ " v5.01 Update: Match 10th 2022 - gandamu / Adam Letts\n",
+ "\n",
+ " IPython magic commands replaced by Python code\n",
+ "\n",
" '''\n",
- " )"
- ]
+ " )\n"
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "XTu6AjLyFQUq"
- },
+ "metadata": {},
"source": [
- "#Tutorial"
+ "# Tutorial"
]
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "YR806W0wi3He"
- },
+ "metadata": {},
"source": [
"**Diffusion settings (Defaults are heavily outdated)**\n",
"---\n",
@@ -251,41 +285,56 @@
"`timestep_respacing` | Modify this value to decrease the number of timesteps. | ddim100\n",
"`diffusion_steps` || 1000\n",
"**Diffusion:**\n",
- "`clip_models` | Models of CLIP to load. Typically the more, the better but they all come at a hefty VRAM cost. | ViT-B/32, ViT-B/16, RN50x4"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "_9Eg9Kf5FlfK"
- },
- "source": [
+ "`clip_models` | Models of CLIP to load. Typically the more, the better but they all come at a hefty VRAM cost. | ViT-B/32, ViT-B/16, RN50x4\n",
+ "\n",
"# 1. Set Up"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "qZ3rNuAWAewx"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"#@title 1.1 Check GPU Status\n",
- "!nvidia-smi -L"
- ]
+ "import subprocess\n",
+ "simple_nvidia_smi_display = False#@param {type:\"boolean\"}\n",
+ "if simple_nvidia_smi_display:\n",
+ " #!nvidia-smi\n",
+ " nvidiasmi_output = subprocess.run(['nvidia-smi', '-L'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " print(nvidiasmi_output)\n",
+ "else:\n",
+ " #!nvidia-smi -i 0 -e 0\n",
+ " nvidiasmi_output = subprocess.run(['nvidia-smi'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " print(nvidiasmi_output)\n",
+ " nvidiasmi_ecc_note = subprocess.run(['nvidia-smi', '-i', '0', '-e', '0'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " print(nvidiasmi_ecc_note)"
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "yZsjzwS0YGo6"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"#@title 1.2 Prepare Folders\n",
+ "import subprocess\n",
+ "import sys\n",
+ "import ipykernel\n",
+ "\n",
+ "def gitclone(url):\n",
+ " res = subprocess.run(['git', 'clone', url], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " print(res)\n",
+ "\n",
+ "def pipi(modulestr):\n",
+ " res = subprocess.run(['pip', 'install', modulestr], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " print(res)\n",
+ "\n",
+ "def pipie(modulestr):\n",
+ " res = subprocess.run(['git', 'install', '-e', modulestr], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " print(res)\n",
+ "\n",
+ "def wget(url, outputdir):\n",
+ " res = subprocess.run(['wget', url, '-P', f'{outputdir}'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " print(res)\n",
"\n",
"try:\n",
" from google.colab import drive\n",
@@ -338,20 +387,18 @@
"\n",
"# libraries = f'{root_path}/libraries'\n",
"# createPath(libraries)"
- ]
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "JmbrcrhpBPC6"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"#@title ### 1.3 Install and import dependencies\n",
"\n",
"from os.path import exists as path_exists\n",
+ "import pathlib, shutil\n",
"\n",
"if not is_colab:\n",
" # If running locally, there's a good chance your env will need this in order to not crash upon np.matmul() or similar operations.\n",
@@ -373,23 +420,31 @@
"model_secondary_downloaded = False\n",
"\n",
"if is_colab:\n",
- " !git clone https://github.com/openai/CLIP\n",
- " # !git clone https://github.com/facebookresearch/SLIP.git\n",
- " !git clone https://github.com/crowsonkb/guided-diffusion\n",
- " !git clone https://github.com/assafshocher/ResizeRight.git\n",
- " !pip install -e ./CLIP\n",
- " !pip install -e ./guided-diffusion\n",
- " !pip install lpips datetime timm\n",
- " !apt install imagemagick\n",
- " !git clone https://github.com/isl-org/MiDaS.git\n",
- " !git clone https://github.com/alembics/disco-diffusion-dev.git\n",
+ " gitclone(\"https://github.com/openai/CLIP\")\n",
+ " #gitclone(\"https://github.com/facebookresearch/SLIP.git\")\n",
+ " gitclone(\"https://github.com/crowsonkb/guided-diffusion\")\n",
+ " gitclone(\"https://github.com/assafshocher/ResizeRight.git\")\n",
+ " pipie(\"./CLIP\")\n",
+ " pipie(\"./guided-diffusion\")\n",
+ " multipip_res = subprocess.run(['pip', 'install', 'lpips', 'datetime', 'timm', 'ftfy'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " print(multipip_res)\n",
+ " subprocess.run(['apt', 'install', 'imagemagick'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " gitclone(\"https://github.com/isl-org/MiDaS.git\")\n",
+ " gitclone(\"https://github.com/alembics/disco-diffusion.git\")\n",
+ " pipi(\"pytorch-lightning\")\n",
+ " pipi(\"omegaconf\")\n",
+ " pipi(\"einops\")\n",
" # Rename a file to avoid a name conflict..\n",
- " !mv MiDaS/utils.py MiDaS/midas_utils.py\n",
- " !cp disco-diffusion-dev/disco_xform_utils.py disco_xform_utils.py\n",
+ " try:\n",
+ " os.rename(\"MiDaS/utils.py\", \"MiDaS/midas_utils.py\")\n",
+ " shutil.copyfile(\"disco-diffusion/disco_xform_utils.py\", \"disco_xform_utils.py\")\n",
+ " except:\n",
+ " pass\n",
"\n",
- "!mkdir model\n",
+ "if not path_exists(f'{model_path}'):\n",
+ " pathlib.Path(model_path).mkdir(parents=True, exist_ok=True)\n",
"if not path_exists(f'{model_path}/dpt_large-midas-2f21e586.pt'):\n",
- " !wget https://github.com/intel-isl/DPT/releases/download/1_0/dpt_large-midas-2f21e586.pt -P {model_path}\n",
+ " wget(\"https://github.com/intel-isl/DPT/releases/download/1_0/dpt_large-midas-2f21e586.pt\", out=model_path)\n",
"\n",
"import sys\n",
"import torch\n",
@@ -402,8 +457,9 @@
" torch.version.cuda.replace(\".\",\"\"),\n",
" f\"_pyt{pyt_version_str}\"\n",
" ])\n",
- " !pip install fvcore iopath\n",
- " !pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html\n",
+ " multipip_res = subprocess.run(['pip', 'install', 'fvcore', 'iopath'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " print(multipip_res)\n",
+ " subprocess.run(['pip', 'install', '--no-index', '--no-cache-dir', 'pytorch3d', '-f', f'https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
"\n",
"# sys.path.append('./SLIP')\n",
"sys.path.append('./ResizeRight')\n",
@@ -443,10 +499,10 @@
"\n",
"#SuperRes\n",
"if is_colab:\n",
- " !git clone https://github.com/CompVis/latent-diffusion.git\n",
- " !git clone https://github.com/CompVis/taming-transformers\n",
- " !pip install -e ./taming-transformers\n",
- " !pip install ipywidgets omegaconf>=2.0.0 pytorch-lightning>=1.0.8 torch-fidelity einops wandb\n",
+ " gitclone(\"https://github.com/CompVis/latent-diffusion.git\")\n",
+ " gitclone(\"https://github.com/CompVis/taming-transformers\")\n",
+ " pipie(\"./taming-transformers\")\n",
+ " pipi(\"ipywidgets omegaconf>=2.0.0 pytorch-lightning>=1.0.8 torch-fidelity einops wandb\")\n",
"\n",
"#SuperRes\n",
"import ipywidgets as widgets\n",
@@ -455,20 +511,22 @@
"sys.path.append('./taming-transformers')\n",
"from taming.models import vqgan # checking correct import from taming\n",
"from torchvision.datasets.utils import download_url\n",
+ "\n",
"if is_colab:\n",
- " %cd '/content/latent-diffusion'\n",
+ " os.chdir('/content/latent-diffusion')\n",
"else:\n",
- " %cd 'latent-diffusion'\n",
+ " #os.chdir('latent-diffusion')\n",
+ " sys.path.append('latent-diffusion')\n",
"from functools import partial\n",
"from ldm.util import instantiate_from_config\n",
"from ldm.modules.diffusionmodules.util import make_ddim_sampling_parameters, make_ddim_timesteps, noise_like\n",
"# from ldm.models.diffusion.ddim import DDIMSampler\n",
"from ldm.util import ismap\n",
"if is_colab:\n",
- " %cd '/content'\n",
+ " os.chdir('/content')\n",
" from google.colab import files\n",
"else:\n",
- " %cd $PROJECT_DIR\n",
+ " os.chdir(f'{PROJECT_DIR}')\n",
"from IPython.display import Image as ipyimg\n",
"from numpy import asarray\n",
"from einops import rearrange, repeat\n",
@@ -481,11 +539,11 @@
"# AdaBins stuff\n",
"if USE_ADABINS:\n",
" if is_colab:\n",
- " !git clone https://github.com/shariqfarooq123/AdaBins.git\n",
+ " gitclone(\"https://github.com/shariqfarooq123/AdaBins.git\")\n",
" if not path_exists(f'{model_path}/AdaBins_nyu.pt'):\n",
- " !wget https://cloudflare-ipfs.com/ipfs/Qmd2mMnDLWePKmgfS8m6ntAg4nhV5VkUyAydYBp8cWWeB7/AdaBins_nyu.pt -P {model_path}\n",
- " !mkdir pretrained\n",
- " !cp -P {model_path}/AdaBins_nyu.pt pretrained/AdaBins_nyu.pt\n",
+ " wget(\"https://cloudflare-ipfs.com/ipfs/Qmd2mMnDLWePKmgfS8m6ntAg4nhV5VkUyAydYBp8cWWeB7/AdaBins_nyu.pt\", out=model_path)\n",
+ " pathlib.Path(\"pretrained\").mkdir(parents=True, exist_ok=True)\n",
+ " shutil.copyfile(f\"{model_path}/AdaBins_nyu.pt\", \"pretrained/AdaBins_nyu.pt\")\n",
" sys.path.append('./AdaBins')\n",
" from infer import InferenceHelper\n",
" MAX_ADABINS_AREA = 500000\n",
@@ -498,16 +556,16 @@
"if torch.cuda.get_device_capability(DEVICE) == (8,0): ## A100 fix thanks to Emad\n",
" print('Disabling CUDNN for A100 gpu', file=sys.stderr)\n",
" torch.backends.cudnn.enabled = False"
- ]
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "BLk3J0h3MtON"
- },
- "outputs": [],
+ "metadata": {},
"source": [
+ "#@title ### 1.4 Define Midas functions\n",
+ "\n",
"from midas.dpt_depth import DPTDepthModel\n",
"from midas.midas_net import MidasNet\n",
"from midas.midas_net_custom import MidasNet_small\n",
@@ -606,19 +664,16 @@
" midas_model.to(DEVICE)\n",
"\n",
" print(f\"MiDaS '{midas_model_type}' depth model initialized.\")\n",
- " return midas_model, midas_transform, net_w, net_h, resize_mode, normalization\n"
- ]
+ " return midas_model, midas_transform, net_w, net_h, resize_mode, normalization"
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "FpZczxnOnPIU"
- },
- "outputs": [],
+ "metadata": {},
"source": [
- "#@title 1.4 Define necessary functions\n",
+ "#@title 1.5 Define necessary functions\n",
"\n",
"# https://gist.github.com/adefossez/0646dbe9ed4005480a2407c62aac8869\n",
"\n",
@@ -1365,20 +1420,16 @@
" }\n",
" # print('Settings:', setting_list)\n",
" with open(f\"{batchFolder}/{batch_name}({batchNum})_settings.txt\", \"w+\") as f: #save settings\n",
- " json.dump(setting_list, f, ensure_ascii=False, indent=4)\n",
- " "
- ]
+ " json.dump(setting_list, f, ensure_ascii=False, indent=4)"
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "TI4oAu0N4ksZ"
- },
- "outputs": [],
+ "metadata": {},
"source": [
- "#@title 1.5 Define the secondary diffusion model\n",
+ "#@title 1.6 Define the secondary diffusion model\n",
"\n",
"def append_dims(x, n):\n",
" return x[(Ellipsis, *(None,) * (n - x.ndim))]\n",
@@ -1540,19 +1591,16 @@
" alphas, sigmas = map(partial(append_dims, n=v.ndim), t_to_alpha_sigma(t))\n",
" pred = input * alphas - v * sigmas\n",
" eps = input * sigmas + v * alphas\n",
- " return DiffusionOutput(v, pred, eps)\n"
- ]
+ " return DiffusionOutput(v, pred, eps)"
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "NJS2AUAnvn-D"
- },
- "outputs": [],
+ "metadata": {},
"source": [
- "#@title 1.6 SuperRes Define\n",
+ "#@title 1.7 SuperRes Define\n",
"class DDIMSampler(object):\n",
" def __init__(self, model, schedule=\"linear\", **kwargs):\n",
" super().__init__()\n",
@@ -2074,25 +2122,20 @@
" a.save(filepath)\n",
" return\n",
" print(f'Processing finished!')\n"
- ]
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "CQVtY1Ixnqx4"
- },
+ "metadata": {},
"source": [
"# 2. Diffusion and CLIP model settings"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "Fpbody2NCR7w"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"#@markdown ####**Models Settings:**\n",
"diffusion_model = \"512x512_diffusion_uncond_finetune_008100\" #@param [\"256x256_diffusion_uncond\", \"512x512_diffusion_uncond_finetune_008100\"]\n",
@@ -2139,12 +2182,12 @@
" model_256_downloaded = True\n",
" else: \n",
" print(\"256 Model SHA doesn't match, redownloading...\")\n",
- " !wget --continue {model_256_link} -P {model_path}\n",
+ " wget(model_256_link, out=model_path)\n",
" model_256_downloaded = True\n",
" elif os.path.exists(model_256_path) and not check_model_SHA or model_256_downloaded == True:\n",
" print('256 Model already downloaded, check check_model_SHA if the file is corrupt')\n",
" else: \n",
- " !wget --continue {model_256_link} -P {model_path}\n",
+ " wget(model_256_link, out=model_path)\n",
" model_256_downloaded = True\n",
"elif diffusion_model == '512x512_diffusion_uncond_finetune_008100':\n",
" if os.path.exists(model_512_path) and check_model_SHA:\n",
@@ -2157,12 +2200,12 @@
" model_512_downloaded = True\n",
" else: \n",
" print(\"512 Model SHA doesn't match, redownloading...\")\n",
- " !wget --continue {model_512_link} -P {model_path}\n",
+ " wget(model_512_link, out=model_path)\n",
" model_512_downloaded = True\n",
" elif os.path.exists(model_512_path) and not check_model_SHA or model_512_downloaded == True:\n",
" print('512 Model already downloaded, check check_model_SHA if the file is corrupt')\n",
" else: \n",
- " !wget --continue {model_512_link} -P {model_path}\n",
+ " wget(model_512_link, out=model_path)\n",
" model_512_downloaded = True\n",
"\n",
"\n",
@@ -2178,12 +2221,12 @@
" model_secondary_downloaded = True\n",
" else: \n",
" print(\"Secondary Model SHA doesn't match, redownloading...\")\n",
- " !wget --continue {model_secondary_link} -P {model_path}\n",
+ " wget(model_secondary_link, out=model_path)\n",
" model_secondary_downloaded = True\n",
" elif os.path.exists(model_secondary_path) and not check_model_SHA or model_secondary_downloaded == True:\n",
" print('Secondary Model already downloaded, check check_model_SHA if the file is corrupt')\n",
" else: \n",
- " !wget --continue {model_secondary_link} -P {model_path}\n",
+ " wget(model_secondary_link, out=model_path)\n",
" model_secondary_downloaded = True\n",
"\n",
"model_config = model_and_diffusion_defaults()\n",
@@ -2247,7 +2290,7 @@
"if SLIPB16:\n",
" SLIPB16model = SLIP_VITB16(ssl_mlp_dim=4096, ssl_emb_dim=256)\n",
" if not os.path.exists(f'{model_path}/slip_base_100ep.pt'):\n",
- " !wget https://dl.fbaipublicfiles.com/slip/slip_base_100ep.pt -P {model_path}\n",
+ " wget(\"https://dl.fbaipublicfiles.com/slip/slip_base_100ep.pt\", out=model_path)\n",
" sd = torch.load(f'{model_path}/slip_base_100ep.pt')\n",
" real_sd = {}\n",
" for k, v in sd['state_dict'].items():\n",
@@ -2261,7 +2304,7 @@
"if SLIPL16:\n",
" SLIPL16model = SLIP_VITL16(ssl_mlp_dim=4096, ssl_emb_dim=256)\n",
" if not os.path.exists(f'{model_path}/slip_large_100ep.pt'):\n",
- " !wget https://dl.fbaipublicfiles.com/slip/slip_large_100ep.pt -P {model_path}\n",
+ " wget(\"https://dl.fbaipublicfiles.com/slip/slip_large_100ep.pt\", out=model_path)\n",
" sd = torch.load(f'{model_path}/slip_large_100ep.pt')\n",
" real_sd = {}\n",
" for k, v in sd['state_dict'].items():\n",
@@ -2273,26 +2316,21 @@
" clip_models.append(SLIPL16model)\n",
"\n",
"normalize = T.Normalize(mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711])\n",
- "lpips_model = lpips.LPIPS(net='vgg').to(device)"
- ]
+ "lpips_model = lpips.LPIPS(net='vgg').to(device)\n"
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "kjtsXaszn-bB"
- },
+ "metadata": {},
"source": [
"# 3. Settings"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "U0PwzFZbLfcy"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"#@markdown ####**Basic Settings:**\n",
"batch_name = 'TimeToDisco' #@param{type: 'string'}\n",
@@ -2330,25 +2368,20 @@
"#Make folder for batch\n",
"batchFolder = f'{outDirPath}/{batch_name}'\n",
"createPath(batchFolder)\n"
- ]
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "CnkTNXJAPzL2"
- },
+ "metadata": {},
"source": [
- "###Animation Settings"
+ "### Animation Settings"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "djPY2_4kHgV2"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"#@markdown ####**Animation Mode:**\n",
"animation_mode = 'None' #@param ['None', '2D', '3D', 'Video Input'] {type:'string'}\n",
@@ -2372,11 +2405,13 @@
" createPath(videoFramesFolder)\n",
" print(f\"Exporting Video Frames (1 every {extract_nth_frame})...\")\n",
" try:\n",
- " !rm {videoFramesFolder}/*.jpg\n",
+ " for f in pathlib.Path(f'{videoFramesFolder}').glob('*.jpg'):\n",
+ " f.unlink()\n",
" except:\n",
" print('')\n",
" vf = f'\"select=not(mod(n\\,{extract_nth_frame}))\"'\n",
- " !ffmpeg -i {video_init_path} -vf {vf} -vsync vfr -q:v 2 -loglevel error -stats {videoFramesFolder}/%04d.jpg\n",
+ " subprocess.run(['ffmpeg', '-i', f'{video_init_path}', '-vf', f'{vf}', '-vsync', 'vfr', '-q:v', '2', '-loglevel', 'error', '-stats', f'{videoFramesFolder}/%04d.jpg'], stdout=subprocess.PIPE).stdout.decode('utf-8')\n",
+ " #!ffmpeg -i {video_init_path} -vf {vf} -vsync vfr -q:v 2 -loglevel error -stats {videoFramesFolder}/%04d.jpg\n",
"\n",
"\n",
"#@markdown ---\n",
@@ -2654,14 +2689,14 @@
" translation_z = float(translation_z)\n",
" rotation_3d_x = float(rotation_3d_x)\n",
" rotation_3d_y = float(rotation_3d_y)\n",
- " rotation_3d_z = float(rotation_3d_z)"
- ]
+ " rotation_3d_z = float(rotation_3d_z)\n"
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "u1VHzHvNx5fd"
- },
+ "metadata": {},
"source": [
"### Extra Settings\n",
" Partial Saves, Diffusion Sharpening, Advanced Settings, Cutn Scheduling"
@@ -2669,12 +2704,7 @@
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "lCLMxtILyAHA"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"#@markdown ####**Saving:**\n",
"\n",
@@ -2745,27 +2775,22 @@
"cut_overview = \"[12]*400+[4]*600\" #@param {type: 'string'} \n",
"cut_innercut =\"[4]*400+[12]*600\"#@param {type: 'string'} \n",
"cut_ic_pow = 1#@param {type: 'number'} \n",
- "cut_icgray_p = \"[0.2]*400+[0]*600\"#@param {type: 'string'} \n",
- "\n"
- ]
+ "cut_icgray_p = \"[0.2]*400+[0]*600\"#@param {type: 'string'}\n"
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "XIwh5RvNpk4K"
- },
+ "metadata": {},
"source": [
- "###Prompts\n",
+ "### Prompts\n",
"`animation_mode: None` will only use the first set. `animation_mode: 2D / Video` will run through them per the set frames and hold on the last one."
]
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "BGBzhk3dpcGO"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"text_prompts = {\n",
" 0: [\"A beautiful painting of a singular lighthouse, shining its light across a tumultuous sea of blood by greg rutkowski and thomas kinkade, Trending on artstation.\", \"yellow color scheme\"],\n",
@@ -2774,26 +2799,21 @@
"\n",
"image_prompts = {\n",
" # 0:['ImagePromptsWorkButArentVeryGood.png:2',],\n",
- "}"
- ]
+ "}\n"
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "Nf9hTc8YLoLx"
- },
+ "metadata": {},
"source": [
"# 4. Diffuse!"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "LHLiO56OfwgD"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"#@title Do the Run!\n",
"#@markdown `n_batches` ignored with animation modes.\n",
@@ -2961,26 +2981,21 @@
"finally:\n",
" print('Seed used:', seed)\n",
" gc.collect()\n",
- " torch.cuda.empty_cache()"
- ]
+ " torch.cuda.empty_cache()\n"
+ ],
+ "outputs": [],
+ "execution_count": null
},
{
"cell_type": "markdown",
- "metadata": {
- "id": "EZUg3bfzazgW"
- },
+ "metadata": {},
"source": [
"# 5. Create the video"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "metadata": {
- "cellView": "form",
- "id": "HV54fuU3pMzJ"
- },
- "outputs": [],
+ "metadata": {},
"source": [
"# @title ### **Create video**\n",
"#@markdown Video file will save in the same folder as your images.\n",
@@ -3056,27 +3071,16 @@
" # mp4 = open(filepath,'rb').read()\n",
" # data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
" # display.HTML(f'')"
- ]
+ ],
+ "outputs": [],
+ "execution_count": null
}
],
"metadata": {
- "accelerator": "GPU",
- "colab": {
- "collapsed_sections": [
- "1YwMUyt9LHG1",
- "XTu6AjLyFQUq",
- "_9Eg9Kf5FlfK",
- "CnkTNXJAPzL2",
- "u1VHzHvNx5fd"
- ],
- "machine_shape": "hm",
- "name": "Disco Diffusion v4.1 [w/ Video Inits, Recovery & DDIM Sharpen].ipynb",
- "private_outputs": true,
- "provenance": [],
- "include_colab_link": true
- },
+ "anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
+ "language": "python",
"name": "python3"
},
"language_info": {
@@ -3089,9 +3093,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.7"
+ "version": "3.6.1"
}
},
"nbformat": 4,
- "nbformat_minor": 0
-}
+ "nbformat_minor": 4
+}
\ No newline at end of file
diff --git a/disco.py b/disco.py
index 4312541..095ce4c 100644
--- a/disco.py
+++ b/disco.py
@@ -1,3 +1,7 @@
+# %%
+"""
"""
+
+
# %%
"""
# Disco Diffusion v5 - Now with 3D animation
@@ -44,7 +48,7 @@ Somnai (https://twitter.com/Somnai_dreams) added Diffusion Animation techniques,
# %%
"""
-Licensed under the MIT License
+@title Licensed under the MIT License
Copyright (c) 2021 Katherine Crowson
@@ -68,9 +72,34 @@ THE SOFTWARE.
--
-@title Licensed under the MIT License
+MIT License
+
+Copyright (c) 2019 Intel ISL (Intel Intelligent Systems Lab)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+--
+
+Licensed under the MIT License
+
+Copyright (c) 2021 Maxwell Ingham
-Copyright (c) 2021 Maxwell Ingham
Copyright (c) 2022 Adam Letts
Permission is hereby granted, free of charge, to any person obtaining a copy
@@ -167,6 +196,11 @@ if skip_for_run_all == False:
v4.92 Update: Feb 20th 2022 - gandamu / Adam Letts
Separated transform code
+
+ v5.01 Update: Match 10th 2022 - gandamu / Adam Letts
+
+ IPython magic commands replaced by Python code
+
'''
)