How to use segmentation_models.pytorch: a beginner's guide

What SMP is and what it is not

SMP is a code library, not a ready-made app. The README documents it as a set of building blocks you call from your own Python code. It is built on PyTorch, so you need PyTorch in your project to use it.

The library offers a high-level API, a range of encoder-decoder architectures such as Unet, Unet++, Segformer and DPT, and pretrained encoders. It also includes popular training metrics and losses such as Dice, Jaccard and Tversky.

• Good fit: Python developers building segmentation models in PyTorch, and researchers comparing architectures and encoders.
• Not a fit: people who want a finished app with no coding, or projects that do not use PyTorch.
• You still need labeled data for your own task. The README covers training with your own dataset and includes examples.

Install segmentation_models.pytorch: clone the repository

The install instructions in the source start with cloning the GitHub repository. The README badges list the minimum Python and PyTorch versions, so check those before you start.

The README excerpt shows a PyPI badge for a package named segmentation-models-pytorch, but the excerpt does not show a pip install command. Check the full README or the documentation at smp.readthedocs.io for the installation steps that match your setup.

• Check the Python and PyTorch minimums shown on the README badges.
• Clone the repository, then follow the install section of the full README or the documentation.

Create your first segmentation model

The README says a segmentation model is a PyTorch torch.nn.Module and can be created in about two lines. The import name is segmentation_models_pytorch, shortened to smp. The example below uses a Unet with a resnet34 encoder that starts from ImageNet weights.

Set in_channels to match your images: 1 for grayscale and 3 for RGB. Set classes to the number of classes in your dataset. The README links to a table of architectures and a table of encoders with their weights.

• encoder_name chooses the encoder, for example mobilenet_v2 or efficientnet-b7.
• encoder_weights='imagenet' starts the encoder from ImageNet pretrained weights.
• in_channels is the number of input channels.
• classes is the number of output channels, which is the number of classes in your data.

Prepare your data with the matching preprocessing

Every encoder has pretrained weights. The README says that preparing your data the same way as during the weights' pre-training may give better results, with higher metric scores and faster convergence. It is not necessary if you train the whole model rather than only the decoder.

Use get_preprocessing_fn from segmentation_models_pytorch.encoders to get the matching function for an encoder and its pretrained weights.

Train the model and run the examples

The README says you can train the model with your favorite framework once it is created. The source does not describe a built-in training loop, so you write or reuse your own training code.

The repository includes example notebooks. They cover training binary segmentation on OxfordPets and CamVid, multiclass segmentation on CamVid, loading pretrained Segformer, DPT and UPerNet models, saving and loading models locally or to the HuggingFace Hub, and exporting to ONNX. Open them from the examples folder or from the Colab links in the README.

• Binary segmentation on OxfordPets (examples/binary_segmentation_intro.ipynb).
• Multiclass segmentation on CamVid (examples/camvid_segmentation_multiclass.ipynb).
• Save and load models, including HuggingFace Hub (examples/save_load_model_and_share_with_hf_hub.ipynb).
• Export to ONNX (examples/convert_to_onnx.ipynb).

Export a trained model to ONNX

The README lists ONNX export as a feature, and the examples include a notebook called convert_to_onnx.ipynb. The source also says the models are torch script, trace and compile friendly.

Use the notebook as your starting point, since the README excerpt does not show the export code itself.