How to Deploy a Machine Learning Model on AWS EC2 in 2026

The global cloud machine learning market is expected to grow from $13.4 billion in 2020 to $117.6 billion by 2027, at a CAGR of 39.2% during the forecast period ^1^. As more organizations adopt ML, the need for reliable and scalable deployment solutions has never been greater. Amazon Web Services (AWS) is a leading cloud provider, with over 200 fully featured services and millions of active customers ^2^.

Within the AWS ecosystem, Elastic Compute Cloud (EC2) is a foundational service that provides secure and resizable compute capacity. EC2 usage grew 28% year-over-year in Q1 2023, making it a popular choice for deploying ML models ^3^. This step-by-step tutorial will walk through the process of deploying a trained machine learning model to an AWS EC2 instance. By the end, you‘ll have a model hosted in the cloud that can be queried via an API to generate predictions. Let‘s dive in!

Step 1: Train and Export Your Model

Before deploying a model, you first need to develop and train it. This typically involves:

1. Collecting and preparing a dataset
2. Choosing an appropriate ML algorithm
3. Training the model on the data
4. Evaluating performance and optimizing hyperparameters
5. Exporting the trained model artifact

The specific steps will vary depending on your use case, preferred ML framework (TensorFlow, PyTorch, scikit-learn, etc.), and programming language. But generally, the output of the training process is a serialized model file (e.g. a .pkl or .h5 file) that encapsulates the model architecture and learned parameters. This is the artifact you‘ll deploy to EC2.

It‘s important to choose a file format that is compatible with your serving framework. For example, if using TensorFlow Serving, you‘ll want to export in the SavedModel format. Consult your framework‘s documentation for recommended export practices.

Step 2: Set Up an EC2 Instance

With a trained model in hand, you‘re ready to provision cloud resources. EC2 offers a wide range of instance types optimized for different use cases. For ML workloads, GPU-powered instances deliver the best performance. Some popular options:

• P3: Up to 8 NVIDIA V100 GPUs, 100 Gbps networking. Ideal for large-scale ML training and high performance computing.
• G4: Up to 4 NVIDIA T4 GPUs, 25 Gbps networking. Well-suited for ML inference and graphics-intensive applications.
• G5: Up to 8 NVIDIA A10G GPUs, 4x the GPU memory of G4. Designed for ML training, 3D rendering, AR/VR, and graphics workstations.

To launch a GPU instance:

1. Open the EC2 console and click "Launch Instance"
2. Select a Deep Learning AMI (more on this later)
3. Choose a GPU-powered instance like g4dn.xlarge
4. Configure storage, networking, and security settings
5. Create or select a key pair for SSH access
6. Launch the instance and wait a few minutes for it to start up
7. Take note of the public IPv4 DNS name

Step 3: Install Dependencies

With your EC2 instance running, connect to it via SSH using the DNS name and .pem key file. Then install the required dependencies.

Many ML frameworks and libraries have complex dependencies that can be tedious to install and manage. To simplify the process, AWS offers Deep Learning AMIs–pre-configured EC2 instances with popular ML tools pre-installed. Supported frameworks include^4^:

• TensorFlow
• PyTorch
• Apache MXNet
• Chainer
• Microsoft Cognitive Toolkit
• Gluon
• Horovod
• Keras
• Scikit-learn
• And more

The AMIs also include NVIDIA drivers, CUDA, and cuDNN for GPU acceleration.

To install additional dependencies not included in your AMI:

1. Update the package manager: sudo apt-get update
2. Install Python and pip: sudo apt-get install python3 python3-pip
3. Install specific versions of libraries, e.g.: pip3 install keras==2.3.0
4. Install WSGI tools like Gunicorn to serve your model API

Step 4: Upload Model to EC2

Next, upload your exported model file to the EC2 instance. For large models, it‘s recommended to use S3 as an intermediate storage layer:

1. Create an S3 bucket and upload your model to it
2. On your EC2 instance, install the AWS CLI: pip3 install awscli
3. Configure the CLI with your access key and secret key
4. Copy the model from S3 to EC2, e.g.: aws s3 cp s3://my-bucket/model.pkl ./model.pkl

Treat serialized model files as sensitive assets, as they may contain proprietary architecture details and trained parameters. Use S3 security best practices like encryption, versioning, and access control to protect them.

Step 5: Create a Web App and API

To expose your model for generating predictions, create a web app with an inference API. Popular frameworks for this include Flask and FastAPI.

Flask Example

import pickle
from flask import Flask, request, jsonify

app = Flask(__name__)

# Load model
with open(‘model.pkl‘, ‘rb‘) as f:
    model = pickle.load(f)

# Define prediction endpoint
@app.route(‘/predict‘, methods=[‘POST‘])
def predict():
    # Get input data from request
    input_data = request.json[‘input_data‘]

    # Make prediction
    prediction = model.predict([input_data])

    # Return prediction as JSON
    return jsonify({‘prediction‘: prediction.tolist()})

if __name__ == ‘__main__‘:
    app.run()

FastAPI Example

import pickle
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

# Load model
with open(‘model.pkl‘, ‘rb‘) as f:
    model = pickle.load(f)

# Define input schema
class ModelInput(BaseModel):
    input_data: List[float]

# Define prediction endpoint
@app.post(‘/predict‘)
def predict(input: ModelInput):
    # Make prediction
    prediction = model.predict([input.input_data])

    # Return prediction
    return {‘prediction‘: prediction.tolist()}

For TensorFlow models, TensorFlow Serving is a performant option:

1. Install TensorFlow Serving on your EC2 instance
2. Create a model configuration file specifying the model name, path, and signature
3. Start the TF Serving server, passing it the configuration file
4. Make prediction requests to the server‘s gRPC or REST endpoints

Thoroughly test your inference endpoint locally before proceeding to the next step.

Step 6: Configure Security

Before allowing public traffic to your inference API, lock down your EC2 instance:

• Create a security group restricting inbound traffic to specific IP ranges and ports (e.g. port 443 for HTTPS)
• Configure a firewall like ufw to allow only necessary incoming connections
• Enable HTTPS by creating an SSL/TLS certificate (e.g. using AWS Certificate Manager) and attaching it to an Application Load Balancer
• Implement secure and scalable API authentication and authorization (e.g. using AWS Cognito or Auth0)

Step 7: Test the Deployed Model

With your inference API deployed and secured, it‘s time to test it end-to-end. Send a POST request to the /predict endpoint, passing in correctly formatted input data. Verify the response contains the expected model output.

Here‘s an example using the Python requests library:

import requests

url = ‘https://my-api.com/predict‘
input_data = [[5.1, 3.5, 1.4, 0.2]]  # Example input

response = requests.post(url, json={‘input_data‘: input_data})
print(response.json())  

Best Practices for EC2 Model Deployments

Plan for Scale

• Load test your inference endpoint to measure its performance under various traffic levels
• Set up autoscaling to automatically adjust instance counts based on load
• Decouple your API frontend from backend model servers to allow independent scaling
• Use container orchestration tools like EKS or ECS for deploying and managing ML microservices at scale

Monitor Performance and Costs

• Monitor CPU, GPU, memory, disk I/O, and network utilization with tools like AWS CloudWatch or Datadog
• Set up alerts for when utilization exceeds predefined thresholds
• Tag EC2 instances and related resources for detailed cost breakdowns and optimization
• Consider using AWS Compute Optimizer to get sizing recommendations for your EC2 instances

Ensure Reliability

• Spread instances across multiple availability zones to minimize downtime from zonal outages
• Implement a blue-green deployment strategy to seamlessly switch to new model versions
• Configure health checks for your instances and automatically replace unhealthy ones
• Regularly backup your model artifacts to S3 or EFS to avoid data loss

Secure Sensitive Data and Models

• Encrypt data in transit (HTTPS) and at rest (e.g. EBS encryption)
• Use EC2 instance roles and AWS IAM to enforce least-privilege access to S3 and other services
• Restrict SSH access and leverage tools like AWS Systems Manager for secure remote management
• Consider deploying models in private VPCs for maximum network isolation

EC2 Deployment Costs

EC2 offers three main pricing options:

1. On-Demand: Pay by the second for the instances you launch, with no long-term commitments. Most expensive but most flexible.
2. Spot Instances: Bid on spare EC2 capacity for up to 90% off On-Demand prices. Instances can be interrupted with a 2-minute warning when EC2 needs the capacity back. Good for fault-tolerant and flexible workloads.
3. Reserved Instances: Get discounts of up to 72% off On-Demand by committing to a 1 or 3 year term. Recommended for predictable and steady-state usage.

Estimating EC2 costs involves considering:

• Instance type and size
• Number of instances
• Storage and data transfer
• Associated services (e.g. Elastic Load Balancer)
• Region

AWS provides a Pricing Calculator to help estimate deployment costs. As an example, deploying two g4dn.xlarge instances with a 400 GB EBS volume and 10 TB/month of data transfer would cost:

Resource	On-Demand Price	Spot Price (70% off)	1yr Reserved (All Upfront)
g4dn.xlarge	$1,036.16/month	$310.85/month	$730/month ($8,755 upfront)
400 GB gp2 EBS	$40/month	$40/month	$40/month
10 TB transfer	$921/month	$921/month	$921/month
Total	$1,997/month	$1,272/month	$1,691/month

Alternatives to EC2

EC2 is one of many options for deploying ML models on AWS. Other services to consider:

• AWS Lambda: Serverless compute for running code without provisioning infra. Well-suited for sporadic and low-latency workloads. Can be 50-80% cheaper than EC2 for low-volume use cases^5^.
• AWS Elastic Kubernetes Service (EKS): Managed Kubernetes for deploying models in containers. Offers portability and orchestration benefits, but requires more setup than EC2. EKS can be up to 50% cheaper than self-managed Kubernetes on EC2^6^.
• AWS SageMaker: End-to-end managed platform for building, training, and deploying models. Higher level of abstraction than EC2, but less control. SageMaker can be 54-76% cheaper than self-managed ML on EC2^7^.
• AWS Elastic Beanstalk: Easy-to-use service for deploying web apps, including ML models. Automatically provisions underlying EC2 and load balancing resources. Can be 28% cheaper than manual deployments^8^.

The best option depends on factors like performance requirements, cost constraints, operational overhead, and architecture complexity. In a benchmark comparing ML inference performance across SageMaker, EC2, and Lambda, SageMaker delivered the lowest latency and highest throughput for most model types^9^.

Conclusion

Deploying ML models on AWS EC2 involves several key steps:

1. Train and export model
2. Provision EC2 instance
3. Install dependencies
4. Upload model
5. Create inference API
6. Configure security
7. Test deployed model

Following best practices around performance, cost, reliability, and security is critical for successful deployments.

While EC2 is a popular choice, it‘s worth evaluating other AWS offerings like Lambda, EKS, SageMaker, and Elastic Beanstalk to find the best fit for your specific use case. The right choice can yield significant performance and cost benefits.

Companies like Intuit, Yelp, and Autodesk have used AWS to deploy ML models at scale, driving billions in incremental revenue^10^. As the cloud ML market continues its rapid growth, more organizations will look to platforms like AWS to bring their models into production quickly and cost-effectively. The future of ML deployment is cloud-first.

References