Guide · Open source
How to use ml-stable-diffusion: Core ML setup on Apple Silicon
Updated · written with AI from the project's own documentation and checked against our catalogue data
ml-stable-diffusion is Apple's open-source repository for running Stable Diffusion, a text-to-image AI model, using Core ML on Apple Silicon devices. It is aimed at developers, engineers and researchers who want to convert Stable Diffusion models to Core ML or add image generation to macOS, iOS and iPadOS apps. This guide walks through the prerequisites, cloning the repository, and the two main parts of the project: the Python package and the Swift package.
ml-stable-diffusionapple-aiml-research/ml-stable-diffusion · ★ 18k · MIT Install options & downloads Quick start
- 1Confirm your hardware and OS. Check that you have an Apple Silicon Mac and macOS 13.1 or later, plus Python 3.8 for conversion.
- 2Clone the repository. Run the git clone command from the README to download the source code.
- 3Read the README setup section. The README is the reference for the installation steps, so follow it for the Python environment and conversion commands it describes.
- 4Convert a Stable Diffusion model to Core ML. Use the python_coreml_stable_diffusion package to convert a PyTorch Stable Diffusion model into Core ML format.
- 5Generate a test image in Python. Run image generation through Hugging Face diffusers with the converted model to confirm the output works.
- 6Add the Swift package to Xcode. In an Xcode 14.3 or later project, add the StableDiffusion Swift package as a dependency and point it at your Core ML model files.
Check the system requirements before you start
The README lists different requirements depending on what you are doing. For model conversion, you need macOS 13.1 or later and Python 3.8, with coremltools 7.0. To build the Swift package in an Xcode project, you need macOS 13.1, Xcode 14.3 and Swift 5.8.
- Model conversion: macOS 13.1, Python 3.8, coremltools 7.0
- Project build: macOS 13.1, Xcode 14.3, Swift 5.8
- Target runtime: macOS 13.1 and iPadOS/iOS 16.2 (macOS 14.0 and iPadOS/iOS 17.0 for the memory improvements)
- Target hardware: M1 for Mac and iPad, A14 for iPhone
Clone the repository
Download the source code with git. The README gives this command for cloning the repository, and the project is then built from the cloned folder.
git clone https://github.com/apple/ml-stable-diffusion.gitUnderstand the two parts of the repository
The repository has two components. The python_coreml_stable_diffusion package converts PyTorch models to Core ML format and can generate images in Python through Hugging Face diffusers. The StableDiffusion Swift package is added to an Xcode project as a dependency so your app can run image generation. The Swift package relies on the Core ML model files produced by the Python package, so conversion comes first.
- python_coreml_stable_diffusion: convert models and generate images in Python
- StableDiffusion: Swift package for Xcode projects that loads the Core ML models
Review the performance benchmarks with care
The README includes latency and speed tables for iPhone, iPad and Mac hardware, using specific model versions at 512x512 and 768x768 output. The benchmarks use 20 inference steps and were run by Apple and Hugging Face on iOS 17.0, iPadOS 17.0 and macOS 14.0 beta builds in 2023. Results depend on the model version, the --compute-unit and --attention-implementation settings, and device conditions such as thermal state, so treat the numbers as reference points rather than guarantees for your device.
- Benchmarks use CPU_AND_NE compute units in the tables shown
- Some runs enabled the reduceMemory option, which adds up to 2 seconds of latency
What people use it for
- Adding on-device image generation to a macOS or iOS app built in Xcode
- Converting a Stable Diffusion model to Core ML for on-device inference
- Testing how diffusion models perform on Apple Silicon hardware such as M1 Macs and iPads
- Generating images in Python with Hugging Face diffusers using a Core ML model
Frequently asked questions
Is ml-stable-diffusion free to use?
The repository is released under the MIT licence, as listed in its details. You can read the licence file in the repository to confirm the terms that apply to your project.
What do I need to run ml-stable-diffusion?
For model conversion the README lists macOS 13.1 or later and Python 3.8 with coremltools 7.0. Building the Swift package requires Xcode 14.3 or later and Swift 5.8.
Does ml-stable-diffusion work on Intel Macs or Windows?
No. The project targets Apple Silicon and Core ML, so it is not intended for Intel Macs or non-Apple platforms.
Can I use ml-stable-diffusion without writing code?
Not really. The README describes a developer package and a Swift project integration, so it is aimed at people who are comfortable with code.
What is the difference between this and diffusers, ComfyUI or InvokeAI?
ml-stable-diffusion focuses on converting models to Core ML and running them on Apple devices, and it uses Hugging Face diffusers for Python generation. Tools like ComfyUI, InvokeAI and stable-diffusion-webui are general interfaces for generating images; this guide does not compare their feature sets in detail.
Which Stable Diffusion model versions are supported?
The benchmarks in the README use specific versions, such as stable-diffusion-2-1-base and stable-diffusion-xl-base-1.0. The README does not describe current model support, so check the repository for the newest versions before you rely on one.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use ml-stable-diffusion if you are a developer or researcher who wants to run Stable Diffusion with Core ML on Apple Silicon, or add image generation to an Apple app. If you just want to generate images without coding, or you need the newest model versions or non-Apple support, pick a general tool such as diffusers, ComfyUI or InvokeAI instead.
Install options & downloads