Vaclav Kosar's face photo
Vaclav Kosar
Software And Machine Learning Blog

BentoML vs Cortex - The ML serving tool showdown

11 May 2020

To find the best ML model serving tool, this post compares major open-source platforms BentoML and Cortex.

BentoML and Cortex logo

Note, however, that I have much more experience with Cortex.

Common

  • both deploy and serve models via API
  • both support major ML frameworks (TensorFlow, Pytorch)
  • both have good documentation

Comparison

Feature BentoML.org Cortex.dev
Language Python - easier to modify, integrate? Go
Deployment Delegated to other tools: Docker, Kubernetes, AWS Lambda, SageMaker, GCP, Azule ML and more. Works currently only with local Docker and AWS EKS (GCP is planned)
Service configuration via decorators directly in Python code. via YAML files
Service packaging and distribution Can be packaged, saved via Python command to management repository with a web dashboard or PyPI Packaging only via Docker images without explicit support
Horizontal scaling Configured separately in other clustering tools. Working on an opinionated Kubernetes deployment. Is part of Cortex configs. But is thus less flexible (private cloud, HTTPS deploy requires custom scripts)
User interface CLI, Web UI CLI
Metrics Prometheus metrics AWS EKS metrics
API Auto-Docs Swagger/OpenAPI N/A
User support Responsive unpaid Slack Channel, but Slack is not the best tool for support Very responsive Gitter
Suggest anything else?

Discussions