How I got this repo (entirely) running with pip (no conda) on M1 Mac

[stillmatic]

Update: everything is working great

1. Follow instructions in https://replicate.com/blog/run-stable-diffusion-on-m1-mac to get your environment setup, but of course clone this repo instead.
2. You should use Python 3.10, not the suggested 3.9 (there is a bug with TypingAlias, as discussed in this thread)
3. Apply chore: update requirements.txt to run on m1 w/pip #337 or check out my fork directly to install the updates.
4. pip install -r requirements.txt; pip install -e .
5. PYTORCH_ENABLE_MPS_FALLBACK=1 python scripts/dream.py --full_precision -A plms --web

You should see a warning >> cuda not available, using device mps - this is good! On fully loaded M1 Max, this is 1.5 it/s on keuler or 3.91it/s on PLMS.

---

I got various errors attempting to run the main repo as is (commit 3ee82d8), with pip install instead of conda install. this laptop doesn't have conda on it at all.

workaround is to setup a python env with 3.10 and update requirements.txt

-numpy==1.19.2
+numpy==1.23.1
+--pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu
 omegaconf==2.1.1
-opencv-python==4.1.2.30
+opencv-python==4.6.0.66

also need to run pip install -e . to find ldm.

this enables me to run scripts and get output:

python ./scripts/orig_scripts/txt2img.py --prompt "ocean" --ddim_steps 5 --n_samples 1 --n_iter 1

✨ ✨ ✨ ✨ (output isn't great, but it works!)

however, dream.py has trouble running, even with the full_precision flag. that shows some errors like:

RuntimeError: expected scalar type BFloat16 but found Float

searching issues, I see that this is possibly due to hardcoded assumptions of CPU vs GPU (see #44 (comment)), internally somewhere the code expects a half-precision for GPU vs full-precision for CPU. since other scripts work but dream.py doesn't work, is there somewhere in dream.py with similar hard coded assumptions that we need to update? -- this is fixed in #319

---

everything runs great now on M1, though I have to set env var PYTORCH_ENABLE_MPS_FALLBACK=1. so, running

PYTORCH_ENABLE_MPS_FALLBACK=1 python scripts/dream.py --full_precision -A plms --web

works great!

[ricardobeat]

Repo was working before the 'toffaletti-dream-m1 branch was merged, now I also got errors in master:

cannot import name 'TypeAlias' from 'typing'

The typing package only had TypeAlias added in 3.10. Looks like this branch expects python 3.10 but the conda environment has not been upgraded to match.

[stillmatic]

@ricardobeat would updating that import to use typing_extensions help, it should backport TypeAlias

though, yes, it is quite confusing if you should use 3.9 or 3.10, just generally. I tried to use 3.9 and ran into trouble installing opencv-python, but things are mostly working with 3.10 now.

[jordanfbrown]

I ran into the TypeAlias issue on Python 3.9 and importing it from typing_extensions got me past that issue:

from typing import Optional, Callable
from typing_extensions import TypeAlias

[yousifa]

I ran into the TypeAlias issue on Python 3.9 and importing it from typing_extensions got me past that issue:

from typing import Optional, Callable
from typing_extensions import TypeAlias

Same. Will submit PR for this.

[donut]

from typing import Optional, Callable
from typing_extensions import TypeAlias

@jordanfbrown Where should I put this? I tried at various places in [scripts/dream.py] to no avail.

[stillmatic]

for the TypeAlias issue - could any of you post a fuller stacktrace? I actually don't even see TypeAlias in the main repo at all (and I am on Python 3.10, running txt2img.py without issue).

[yousifa]

its in the k-diffusion dependency in "src/k-diffusion/k_diffusion/sampling.py"

[jordanfbrown]

As @yousifa mentioned, it's coming from this line: https://github.com/Birch-san/k-diffusion/blob/mps/k_diffusion/sampling.py#L10

Full stack trace on Python 3.9:

Traceback (most recent call last):
  File "/Users/jordan/stable-diffusion/scripts/dream.py", line 553, in <module>
    main()
  File "/Users/jordan/stable-diffusion/scripts/dream.py", line 41, in main
    from ldm.simplet2i import T2I
  File "/Users/jordan/stable-diffusion/ldm/simplet2i.py", line 29, in <module>
    from ldm.models.diffusion.ksampler import KSampler
  File "/Users/jordan/stable-diffusion/ldm/models/diffusion/ksampler.py", line 2, in <module>
    import k_diffusion as K
  File "/Users/jordan/stable-diffusion/src/k-diffusion/k_diffusion/__init__.py", line 1, in <module>
    from . import augmentation, config, evaluation, external, gns, layers, models, sampling, utils
  File "/Users/jordan/stable-diffusion/src/k-diffusion/k_diffusion/external.py", line 6, in <module>
    from . import sampling, utils
  File "/Users/jordan/stable-diffusion/src/k-diffusion/k_diffusion/sampling.py", line 10, in <module>
    from typing import Optional, Callable, TypeAlias
ImportError: cannot import name 'TypeAlias' from 'typing' (/Users/jordan/opt/miniconda3/envs/ldm/lib/python3.9/typing.py)

[yousifa]

It seems that this typing import is being added when installing the dependency as it is not in the k-diffusion repo either in sampling.py

[stillmatic]

opened PR in the upstream repo: Birch-san/k-diffusion#1

could also try replacing the dep in environment-mac.yaml with my fork

- -e git+https://github.com/Birch-san/k-diffusion.git@mps#egg=k-diffusion
+ -e git+https://github.com/stillmatic/k-diffusion.git@patch-1#egg=k-diffusion

[yousifa]

maybe we should add a patch to be applied after dependency install on this repo and pin k-diffusion, its at head right now

[irrg]

opened PR in the upstream repo: Birch-san/k-diffusion#1

could also try replacing the dep in environment-mac.yaml with my fork

- -e git+https://github.com/Birch-san/k-diffusion.git@mps#egg=k-diffusion
+ -e git+https://github.com/stillmatic/k-diffusion.git@patch-1#egg=k-diffusion

@stillmatic with this edit, I just get an infinite spinner on Installing pip dependencies:

my PIP block in environment-mac.yaml looks like this:

  - pip:
    - invisible-watermark
    - test-tube
    - tokenizers
    - torch-fidelity
    - -e git+https://github.com/huggingface/diffusers.git@v0.2.4#egg=diffusers
    - -e git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers
    - -e git+https://github.com/openai/CLIP.git@main#egg=clip
    - -e git+https://github.com/stillmatic/k-diffusion.git@patch-1#egg=k-diffusion    
    - -e .

PEBKAC? Did I mess up the edit?

[yousifa]

opened PR in the upstream repo: Birch-san/k-diffusion#1
could also try replacing the dep in environment-mac.yaml with my fork

- -e git+https://github.com/Birch-san/k-diffusion.git@mps#egg=k-diffusion
+ -e git+https://github.com/stillmatic/k-diffusion.git@patch-1#egg=k-diffusion

@stillmatic with this edit, I just get an infinite spinner on Installing pip dependencies:

my PIP block in environment-mac.yaml looks like this:

  - pip:
    - invisible-watermark
    - test-tube
    - tokenizers
    - torch-fidelity
    - -e git+https://github.com/huggingface/diffusers.git@v0.2.4#egg=diffusers
    - -e git+https://github.com/CompVis/taming-transformers.git@master#egg=taming-transformers
    - -e git+https://github.com/openai/CLIP.git@main#egg=clip
    - -e git+https://github.com/stillmatic/k-diffusion.git@patch-1#egg=k-diffusion    
    - -e .

PEBKAC? Did I mess up the edit?

Just tried it. Worked for me. Maybe you messed up the formatting in the file.

[jeffomatic]

with this edit, I just get an infinite spinner on Installing pip dependencies:

I'm also observing this behavior as well.