Vaclav Kosar's face photo
Vaclav Kosar
Software And Machine Learning Blog

BentoML vs Cortex - ML Serving Showdown

To find the best model serving tool, compare open-source MLOps platforms BentoML and Cortex.

BentoML and Cortex logo

Updated 2021-02-21. Do you need a simple way to train and host your machine learning AI models in the cloud? Here is my experience with Cortex Labs’s Cortex. My view on BentoML is based on cursory overview of their documentation. Checkout also AWS App Runner option at the end.


  • both deploy and serve models via API
  • both support major ML frameworks (TensorFlow, Pytorch)
  • both have good documentation


Language fully Python - easier to modify? Go & Python wrapper (Updated)
Deployment Delegated to other tools: Docker, Kubernetes, AWS Lambda, SageMaker, GCP, Azule ML and more. Works currently only with local Docker and AWS EKS, GCP (Updated)
Service configuration from Python from Python (Updated)
Service packaging and distribution Can be packaged, saved via Python command to management repository with a web dashboard or PyPI Packaging only via Docker images without explicit support
Horizontal scaling Configured separately in other clustering tools. Working on an opinionated Kubernetes deployment. Configurable in Cortex. May be less flexible (private cloud deploy may require custom scripts)
User interface CLI, Web UI CLI
Metrics Prometheus metrics Prometheus metrics (Updated)
API Auto-Docs Swagger/OpenAPI N/A
User support Responsive unpaid Slack Channel, but Slack is not the best tool for support Very responsive Gitter and now Slack
Suggest anything else?

My Experience with Cortex

Here is a blog post on Cortex use at GLAMI. It is a bit outdated take as Cortex now has its own wrapper. Consider using this Cortex client for Python, which is a Python wrapper around Cortex CLI that we use at GLAMI for MLOps. It has couple of extra features, that keep us using it for now.


BentoML vs Cortex - ML Serving Showdown

AWS App Runner

If you need super simple deployment of CPU only applications with auto-scaling you can consider using AWS App Runner. You just fill in your source code repository and you app gets hosted in the selected region on your domain with auto certificate renewal.

External Discussions

11 May 2020

Privacy Policy How many days left in this quarter?