What is the rate limit for bots? A deep dive

Hey there! Terry here, back with another in-depth tech explainer for you. Today we‘re going to dive deep into the world of rate limiting, an essential technique used by APIs and online services to prevent abuse.

Grab a coffee, get comfy, and let‘s geek out!

An intro to rate limiting

In the simplest terms, rate limiting sets a threshold on the number of requests you can make to a service within a certain timeframe – say 10 requests per second, or 1500 requests per day. Once you hit that threshold, you‘ll get cut off until the time period resets.

It‘s a bit like the express checkout at the supermarket – they only allow a certain number of people through at once to keep things running smoothly for everyone.

Now APIs and online services work the same way. Rate limiting prevents any single user (or bot) from overwhelming things and impacting everyone else on the platform.

Some common examples:

• Twitter limits you to 300 tweets per day.
• Reddit restricts you to 30 comments per hour in most subreddits.
• Google will cut you off from search after 100 queries per day.

So in summary, rate limits are about sharing resources fairly and preventing system abuse. Pretty simple right?

But there‘s a lot more nuance when we dig deeper…

Why do we need rate limiting anyway?

There‘s a few key reasons why rate limiting is so widely used online:

Preventing abuse

This is the big one. Rate limits allow APIs to control abusive bots or users that want to spam/overload them. Some common abuse tactics include:

• DDoS attacks – flooding servers with traffic to take them offline.
• Scraping data – aggressively pulling as much data as possible from an API.
• Spam requests – repeatedly calling endpoints to fill up databases with junk.

Without rate limits, a single bad actor could easily overwhelm most online services with endless requests.

Fair usage

Rate limits aren‘t just about stopping abuse though – they also ensure fair and equal usage across all users.

Let‘s imagine an API with absolutely no rate limit. User A writes a script to query it 10,000 times per day. User B writes a little app that makes 50 requests per day.

Without a rate limit, User A would squeeze out and impact User B. By applying limits evenly across all users, everyone gets their fair share of requests.

Cost control

This one‘s simple – every request costs the API provider money!

Bandwidth, compute resources and storage all cost money. Rate limits allow providers to control and manage these costs.

It‘s the same reason your phone plan or broadband contract has data limits. Usage costs money, so limits keep things affordable.

Maintaining performance

If an API or service gets overloaded with requests, performance will grind to a halt for everyone.

Rate limits help ensure consistent performance by preventing any single user from hogging resources and slowing things down.

It‘s like an eight lane highway – you can only push so many cars through before traffic jams up. Limits regulate the traffic flow.

So in summary, rate limits promote:

• Abuse prevention
• Fair usage
• Cost control
• Reliable performance

That‘s why they‘re so universal across the web today. But how do they actually work under the hood? Let‘s explore some popular techniques…

Common rate limiting strategies

While the goal is always to restrict traffic, there‘s a few different ways APIs go about implementing rate limits:

Method #1 – Per endpoint limits

This is where each API endpoint or route has its own rate limit applied.

For example, Twitters API has these limits:

• Timeline endpoints – 300 requests per 15 minutes
• Search endpoints – 180 requests per 15 minutes
• User endpoints – 900 requests per 15 minutes

The advantage here is more granular control. You can apply stricter limits on expensive endpoints, while being more relaxed on simple data lookups.

Method #2 – Per user limits

A popular approach is having a pool of requests allocated per user or per API key.

For example, Reddit applies these limits:

• Anonymous users – 60 requests per minute
• OAuth users – 600 requests per 10 minutes

This ensures fair usage across all users. One bad apple can‘t ruin it for everyone.

Method #3 – Global limits

Some APIs will apply a single global rate limit, shared across all endpoints and users.

For example, Google Maps has:

• All APIs – 50 requests per second combined

The advantage here is simplicity. Although it lacks granularity.

Method #4 – Hybrid methods

Many providers use a hybrid approach, combining global limits with per endpoint and per user limits.

This gives both overall protection against spikes, plus more fine-grained control over usage patterns.

Now let‘s look at how these limits actually get implemented…

How rate limiting is enforced

While the policies may differ, these are some popular technical ways rate limits get applied:

In app code

The most straightforward option – add rate limiting logic directly into your app code itself.

Every request runs through a piece of code that checks:

• Is the user under their limit?
• Is the endpoint under its limit?
• Is the whole app under its global limit?

And if any threshold is exceeded, return a 429 Too Many Requests error.

(Although this doesn‘t allow for caching and efficiencies that some other methods do.)

Web server middlewares

For sites and APIs sitting behind a web server like Nginx, rate limiting plugins are popular.

These plugins essentially do the same as in app code, but live at the web server layer, protecting all apps behind them.

Some examples are Nginx Limit Req and Apache mod_ratelimit.

The web server can limit by source IP, request headers, or any other attributes.

API gateways

API gateways like Kong and Apigateway sit in front of apps and act as a proxy. This allows them to easily rate limit incoming requests.

They provide a centralized way to apply policies across multiple apps and services in a standard way.

And they can cache rate limit counters in memory for efficiency and performance.

Cloud services

Most major cloud providers like AWS, GCP and Azure now provide rate limiting services:

• AWS WAF rate limiting
• Google Cloud Armor
• Azure API Management

These make it easy to quickly implement standardized rate limits across resources and apps.

Caching layers

A popular option is using a caching layer like Redis or Memcached to keep count of requests.

As each request comes in, the cache is incremented. Once the threshold is hit, additional requests are rejected until the cache resets.

This is efficient as the cache can be shared across multiple app servers.

So in summary, rate limiting logic can live in multiple places like app code, middlewares, gateways and caches.

What are some typical rate limiting policies?

Now that we understand the how, let‘s look at some real world examples of rate limiting policies used by major APIs:

API	Rate Limit	Notes
Twitter	300 requests per 15 mins	Per authenticated user
Reddit	60 requests per minute	Higher for OAuth apps
GitHub	10,000 requests per hour	Per IP address
Stripe	200 requests per second	Can be increased
Slack	1 request per second	Per workspace
Google Maps	50 requests per second	Global limit
SendGrid	300 requests per second	Per IP address
Cloudflare	1200 requests per 5 minutes	Per IP address

Some key takeaways:

• Stricter limits often apply to anonymous or public usage compared to authenticated requests.
• Granular endpoint limits are common for platform APIs like Twitter, GitHub, Stripe.
• Global limits per second are popular for high volume services like mapping, email.
• Caching providers like Cloudflare and Akamai enforce strict limits.
• Most providers allow verified higher request volumes upon request.

So limits can vary substantially based on the architecture and business model behind each API or service.

Now before moving on, let‘s break down some common terminology:

RPS – Requests per second, a common metric for very high volume APIs.

Origin traffic – Requests coming from your own services vs third parties.

Burst – A higher short term allowance that allows temporary limit breaches.

Throttling – Restricting requests over the limit but not blocking outright.

429 Too Many Requests – The HTTP response code for rate limited requests.

So in summary, policies are tailored for each provider‘s specific needs. Now let‘s look at dealing with limits as an API consumer…

Best practices for using rate limited APIs

Rate limits impact application developers accessing APIs and services. Here‘s some tips for handling restricted providers:

• Monitor usage – Track how close you are to limits in real time to avoid breaches.
• Scale capacity – Use multiple IPs or app instances to distribute load.
• Prioritize requests – Drop non essential calls when approaching limits.
• Queue requests – Queue non urgent requests to smooth out spikes.
• Back off when throttled – Pause and retry if throttled or getting 429s.
• Refactor apps – Reduce calls through caching and code optimization.
• Ask for more – Request higher limits where restrictions impact core usage.
• Read the docs! – Check docs for exact limits and request allocations.

And never assume an API has infinite capacity! Planning for limits will make your app more resilient.

Let‘s recap the key points:

💡 Monitor usage and watch limits closely.
💡 Scale capacity across IPs to raise effective limits.
💡 Prioritize and queue requests to manage peaks.
💡 Back off and retry when hitting 429 errors.
💡 Refactor apps to drive down unnecessary requests.

Now I want to call out one final tip…

Special advice for running bots

If you‘re running bots on restricted platforms like Twitter or Reddit, caution is required!

Those providers design limits specifically to curb bot activity, so stay well clear of any thresholds.

My advice for bot makers:

• Really minimize activity – Only make essential requests and cut out any waste.
• Spread bots across accounts – Distribute load across multiple accounts.
• Use proxy rotation – Rotate through residential proxy IPs to appear more human.
• Slow down interactions – Put real pauses between actions to disguise patterns.
• Anonymous accounts – Use new unlinked accounts regularly to avoid bans.

Basically be a model citizen! Sky high usage on fresh accounts will get flagged quickly.

So tread lightly and mimic real human behaviors as much as possible.

In closing – are you a rate limit expert now?

Phew, that was an epic rate limiting deep dive!

To quickly recap:

• Rate limits allow fair sharing of online resources.
• They prevent abuse and keep services running smoothly.
• Different strategies like per endpoint and per user limits apply.
• Limits can be enforced in app code, APIs, caches and more.
• Policies vary based on each provider‘s infrastructure and business.
• Monitor usage, spread load and retry to avoid breaching.

And most importantly – respect rate limits, and they‘ll respect you back!

So I hope this breakdown has given you a complete overview of rate limiting and how it enables the modern web. Feel free to hit me up on Twitter or LinkedIn if you have any other questions!