Python Requests Library: 2026 Guide

The Python Requests library is one of the most popular libraries for sending HTTP requests in Python. With its simple interface, Requests makes it easy to interact with APIs, scrape websites, and perform other HTTP tasks. In this comprehensive guide, we‘ll cover everything you need to know about using Requests in 2024.

Overview of Requests

Requests is an elegant and simple HTTP library for Python. Some key features include:

• Simple, yet powerful interface for making HTTP requests
• Support for various authentication mechanisms like OAuth, API keys, etc.
• Automatic JSON decoding for JSON responses
• Connection pooling and sessions for performance and faster requests
• Support for streaming large requests without buffering them in memory
• Automatic decompression of gzip/zlib responses
• Timeout and retries to gracefully handle failures
• Works out of the box with Python 2 and 3

Compared to Python‘s built-in urllib/urllib2 libraries, Requests makes sending HTTP requests much simpler with an intuitive API. Requests follows the UNIX philosophy of doing one thing and doing it well. Overall, it‘s a very developer-friendly library.

Installation

Since Requests isn‘t part of the Python standard library, you‘ll need to install it before using it. The recommended way is via pip:

pip install requests

This will install the latest stable release of Requests from PyPI.

Alternatively, you can install from source by cloning the Git repository:

git clone https://github.com/psf/requests.git
cd requests
python setup.py install

After installing, import Requests in your Python script to start using it:

import requests

Making Requests

The Requests library revolves around the requests.request() method. But for convenience, there are also shorthand methods for different HTTP methods like GET, POST, PUT, etc.

Let‘s look at some examples of making requests with Requests.

GET Request

To make a GET request, call requests.get() and pass the URL:

import requests

response = requests.get(‘https://api.github.com/events‘)

This will make a GET request to the specified URL and return a Response object with all the response data.

POST Request

For POST requests, call requests.post() and pass the URL along with any data:

import requests

data = {‘name‘:‘John‘, ‘age‘:30}
response = requests.post(‘https://httpbin.org/post‘, data=data)

This will send a POST request to the URL with the data dictionary serialized as JSON.

PUT Request

Similarly, you can make PUT requests using requests.put():

import requests

data = {‘title‘: ‘New Title‘}
response = requests.put(‘https://httpbin.org/put‘, data=data)

DELETE Request

To send a DELETE request, call requests.delete() and pass the URL:

import requests

response = requests.delete(‘https://httpbin.org/delete‘) 

Adding Parameters

You can add URL query parameters to your requests using the params keyword argument:

import requests

params = {‘page‘: 2, ‘count‘: 25}
response = requests.get(‘https://httpbin.org/get‘, params=params)

This will generate a URL like:

https://httpbin.org/get?page=2&count=25

Adding Headers

To add HTTP headers to your requests, use the headers keyword argument and pass a dictionary:

import requests

headers = {‘User-Agent‘: ‘MyBot‘}
response = requests.get(‘https://httpbin.org/headers‘, headers=headers)

You can add any custom headers like User-Agent, Authorization, Content-Type etc. based on your requirements.

Response Content

The Response object returned by Requests contains all the information about the HTTP response.

To get the response content, access the text attribute:

print(response.text)

This will print the raw response content as a string.

For JSON responses, you can directly access the decoded JSON data using the json() method:

print(response.json())

This will return a dictionary or list from the JSON data.

Response Status Codes

You can check the HTTP status code of the response using the status_code attribute:

print(response.status_code)

Common status codes you may see:

• 200 – Success
• 301 – Redirect
• 400 – Client error like bad request
• 401 – Unauthorized
• 403 – Forbidden
• 404 – Not found
• 500 – Server error

Handling Errors

By default, Requests will raise an exception for HTTP error status codes like 400 or 500.

To handle errors gracefully, you can check the status code manually:

if response.status_code == 200:
    # Success code
elif response.status_code == 404:
    # Not Found Error

Or you can use the raise_for_status() method to raise an exception if the status is an error code:

response.raise_for_status()

This will raise requests.HTTPError if the status is an error code.

Setting Timeouts

You can tell Requests to stop waiting for a response after a given number of seconds using the timeout parameter:

requests.get(‘https://github.com‘, timeout=3)

This will raise a requests.Timeout exception if the request doesn‘t complete within 3 seconds.

Advanced Usage

Now that we‘ve seen the basics, let‘s go over some advanced features of Requests.

Sessions

The Session object allows you to persist parameters across requests. This is useful to avoid repeated authentication steps or saving cookies between requests.

To use sessions:

session = requests.Session()

session.get(‘https://httpbin.org/cookies/set/sessioncookie/123456789‘)
r = session.get(‘https://httpbin.org/cookies‘)

print(r.text)

This will print the session cookie set earlier in the cookies for the second request.

Authentication

Requests supports basic HTTP authentication as well as more complex mechanisms like OAuth.

For basic auth, use the auth parameter with a (username, password) tuple:

requests.get(‘https://api.github.com/user‘, auth=(‘user‘, ‘pass‘))

For OAuth, pass the OAuth token in the Authorization header:

token = ‘xxxxxx‘

requests.get(‘https://api.github.com/user/repos‘, 
            headers={‘Authorization‘: ‘token ‘ + token})

File Uploads

To upload files, pass a file-like object as the files parameter to requests.post():

import requests 

url = ‘https://httpbin.org/post‘
files = {‘file‘: open(‘report.xls‘, ‘rb‘)}

r = requests.post(url, files=files)

This will upload the given file using multipart encoding. You can use a different key name instead of file too.

Downloading Files

To download a file, stream the response content to a file:

import requests

r = requests.get(‘https://example.com/large_file‘, stream=True)

with open(‘downloaded_file‘, ‘wb‘) as fd:
    for chunk in r.iter_content(chunk_size=1024): 
        fd.write(chunk)

This will download the file 1024 bytes at a time and write to disk.

Proxies

To use a proxy for your requests, set the proxies parameter with the proxy URL:

proxies = {
  ‘http‘: ‘http://10.10.1.10:3128‘,
  ‘https‘: ‘http://10.10.1.10:1080‘,
}
requests.get(‘https://example.com‘, proxies=proxies)

You can also use basic auth with your proxy by adding http://user:password@ before the proxy domain/IP.

SSL Certificate Verification

By default, Requests verifies SSL certificates to ensure connections are secure. To disable verification, set verify=False:

requests.get(‘https://example.com‘, verify=False)

However, it‘s strongly recommended to keep SSL verification enabled for security.

If you need to use a custom CA bundle or client-side certificate, you can specify the path:

requests.get(‘https://example.com‘, verify=‘/path/to/custom_ca_bundle‘)

requests.get(‘https://example.com‘, cert=‘/path/to/client.cert‘)

Conclusion

The Requests library makes sending HTTP requests in Python very straightforward. With its simple yet powerful API, built-in JSON handling, authentication, sessions, and other features, it‘s suitable for scraping web content, interacting with APIs, downloading files, and automating web interactions.

Hopefully this guide gave you a comprehensive overview of using Python Requests in 2024 for all your HTTP needs. The official documentation also provides plenty of details and examples for more advanced usage.