Guide · Open source
How to use ONNX Runtime: install, run models and tips
Updated · written with AI from the project's own documentation and checked against our catalogue data
ONNX Runtime is an open-source library from Microsoft that runs machine learning models on many kinds of hardware and operating systems. It is aimed at developers who want to deploy a trained model inside an application or service. This guide explains what the project covers, how to get the code, where the official docs and examples live, and what to consider before you use it.
ONNX Runtimemicrosoft/onnxruntime · ★ 22.2k · MIT Install options & downloads Quick start
- 1Read the README and docs. Start with onnxruntime.ai/docs to confirm the inference or training path that matches your model and hardware.
- 2Clone the repository. Run the clone command from the README to get the source code locally.
- 3Pick a sample repository. Open the inference or training examples repository that matches your goal and follow its code.
- 4Check the release notes. Use the GitHub releases page to see the current and past releases before choosing one.
- 5Test on your target hardware. Run your model on the devices you plan to deploy to, since the README stresses hardware and OS compatibility.
- 6Report problems. Use GitHub Issues for bugs and GitHub Discussions for questions.
What ONNX Runtime does
According to its README, ONNX Runtime has two main features: inference and training. Inference means running a trained model to get predictions. The README says it supports models from deep learning frameworks such as PyTorch and TensorFlow/Keras, as well as classical libraries such as scikit-learn, LightGBM and XGBoost.
The README states that it is compatible with different hardware, drivers and operating systems, and that it uses hardware accelerators where applicable along with graph optimizations and transforms. The stated goal is faster performance and lower costs.
- Inference: run trained models inside an application or service.
- Training: the README says it can speed up training of transformer models on multi-node NVIDIA GPUs with a one-line addition to existing PyTorch training scripts.
Get the source code
The install method listed for this project is a package manager, and the README gives a clone command for the repository. The README does not list prerequisites, so check the official docs for what your platform needs before you start.
git clone https://github.com/microsoft/onnxruntimeFind the docs and sample projects
The README links to the main documentation at onnxruntime.ai/docs, plus YouTube video tutorials. Two companion repositories hold sample code: one for inferencing and one for training. A roadmap page covers upcoming releases.
For releases, the README points to the GitHub releases page rather than listing versions here.
- General information and docs: onnxruntime.ai and onnxruntime.ai/docs
- Inference examples: github.com/microsoft/onnxruntime-inference-examples
- Training examples: github.com/microsoft/onnxruntime-training-examples
- Release roadmap: onnxruntime.ai/roadmap
Choose a workflow that fits the README's focus
The README focuses on production inference and training acceleration. If you have a model already trained in PyTorch, TensorFlow/Keras or a classical library, the inference path is the main use. If you have PyTorch training scripts for transformer models on NVIDIA GPUs, the README's training feature is relevant.
If you only want to experiment in a notebook, this project may be more than you need, since the README does not describe beginner workflows.
Know the limits and get help
The README says the project may collect usage data and send it to Microsoft. The privacy statement is in docs/Privacy.md. For bugs and feature requests, use GitHub Issues; for questions, use GitHub Discussions. Contribution guidelines are in CONTRIBUTING.md.
Comparable projects named in the analysis include TensorRT, OpenVINO, TVM and ncnn. Compare them against your own hardware and model needs.
What people use it for
- Deploying a trained model inside a desktop or server application.
- Running the same model across Windows, Linux, macOS or ARM devices.
- Speeding up inference on available hardware accelerators.
- Testing the README's training acceleration on existing PyTorch scripts for transformer models.
Frequently asked questions
Is ONNX Runtime free to use?
ONNX Runtime is licensed under the MIT License, as stated in the README. Check the LICENSE file in the repository for the full terms.
What do I need to install ONNX Runtime?
The source lists no prerequisites. The README gives a git clone command, and the official docs at onnxruntime.ai/docs cover setup details.
Which models does ONNX Runtime support?
The README says it supports models from PyTorch and TensorFlow/Keras, and classical libraries such as scikit-learn, LightGBM and XGBoost.
What are alternatives to ONNX Runtime?
The analysis lists TensorRT, OpenVINO, TVM and ncnn as comparable projects. Each targets inference on different hardware, so compare them against your setup.
Can ONNX Runtime train models?
The README describes a training feature that can speed up transformer model training on multi-node NVIDIA GPUs with a one-line addition to existing PyTorch training scripts.
Where do I report a bug?
The README asks for bugs and feature requests to be filed as GitHub Issues. General questions go to GitHub Discussions.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
ONNX Runtime suits developers who want to deploy a trained model in an application or service across several operating systems and hardware types. It is less suited to people who only want a beginner notebook workflow, since the README focuses on production inference and training acceleration. If you need a simple experimentation tool or have no programming background, pick something designed for that.
Install options & downloads