How to Send GET Requests With cURL

cURL is a versatile command-line tool that allows you to transfer data to and from servers using various protocols. Besides extracting public data, which can be HTML, text, JSON, images, etc., cURL can also handle sessions and cookies. It can connect with APIs and be easily incorporated with shell commands for automation. All these capabilities make cURL a great tool for web scraping and data extraction. In this comprehensive guide, we‘ll explore how to send GET requests with cURL.

What is a GET Request?

A GET request is an HTTP request method used to retrieve data from a specific resource. For example, when you type a URL into your browser‘s address bar and hit enter, the browser sends a GET request to the server hosting that URL, asking for the associated HTML document.

While other HTTP request methods like POST and PUT can also result in a server sending back HTML, those methods are meant for other purposes. For instance, HTTP POST is used to submit data to be processed, like sending form data. A POST request submits data to be created on the server. In contrast, a GET request simply fetches data – it doesn‘t modify any resources on the server.

Key Characteristics of GET Requests

• GET requests retrieve resources from a server. They do not modify data on the server.
• GET requests can be cached and remain in the browser history.
• GET requests have size limits – URLs can only be so long.
• GET requests send data via query strings in the URL.
• GET requests are idempotent – sending the same request multiple times has no additional effects.

Here‘s a simple example GET request using cURL:

curl https://example.com

This command sends a GET request to the example.com server and prints the HTML document to the terminal.

Now let‘s go through the steps to send various types of GET requests with cURL.

Steps to Send GET Requests With cURL

In this tutorial, we‘ll use httpbin.org, a simple HTTP request testing service that responds with different types of data.

1. Simple GET Request

To send a basic GET request with cURL, specify the URL you want to request after curl:

curl http://httpbin.org/get

This will send a GET request to the /get endpoint and retrieve the response.

If you‘re making a GET request, you can omit the -X GET option since GET is the default HTTP method in cURL.

2. GET Request with Parameters

To send parameters along with a GET request, use the -G option followed by -d to specify the key-value pairs:

curl -G -d "name=John" -d "age=30" http://httpbin.org/get

This sends a GET request to httpbin.org/get with two parameters, name and age. The -G flag tells cURL to use the GET method and append the data to the URL query string.

You can also specify the parameters directly in the URL:

curl ‘http://httpbin.org/get?name=John&age=30‘

Either way works for sending GET parameters with cURL.

3. Include HTTP Headers in Response

By default, cURL only displays the response body. To view the HTTP headers along with the response, use the -i or --include option:

curl -i http://httpbin.org/get

This will print the HTTP status line, headers like Server, Date, Content-Type, etc. along with the response body.

To view only the response headers, not the body, use --head:

curl --head http://httpbin.org/get

4. Get JSON Response

Many modern APIs serve data in JSON format. To tell the server you want JSON, include the -H flag followed by the header:

curl -H "Accept: application/json" http://httpbin.org/get

This adds the HTTP header Accept: application/json to the request, signaling you want JSON back.

• Tip: Whether you get back JSON or not depends on if the API supports JSON responses. The Accept header just informs the server that you prefer JSON.

5. Follow Redirects

To follow redirects returned by the server, use the -L or --location flag:

curl -L http://httpbin.org/redirect-to?url=http://example.com

By default, cURL does not follow redirects from the server. The -L option tells cURL to follow where the server redirects.

6. Send Cookies in GET Request

If you need to send cookies with your GET request, use the -b or --cookie option:

curl -b "session_id=1234; csrftoken=token" http://httpbin.org/cookies

This will send the cookies session_id and csrftoken along with the GET request.

Common cURL Options for GET Requests

Here is a table summarizing some common cURL options for sending GET requests:

Option	Description
-X GET	Specifies GET request method (default)
-i, --include	Includes HTTP response headers in output
--head	Fetch headers only, no response body
-H "Accept: application/json"	Request JSON response format
-L, --location	Follow redirects
-b, --cookie	Send cookie(s)
-d, -G	Send GET parameters
-A, --user-agent	Set user agent string
-v, --verbose	Verbose logging
-o, --output	Write output to file

There are many more options available for fine-tuning your cURL GET requests. Refer to the cURL man pages for complete documentation.

GET Request Examples

Let‘s look at a few examples of sending GET requests with cURL for common use cases:

Scrape a Web Page

To scrape the HTML of a web page, send a simple GET request:

curl https://example.com > example.html

This retrieves the HTML code of example.com and saves it to a file called example.html.

Check API Status

Many APIs provide status endpoints that just return basic info that the API is working. Send a GET request to test:

curl https://api.example.com/status

This should display a JSON response with status info if the API is up.

Search an API

Pass search parameters along with your GET request to query an API:

curl -G -d "term=pizza" -d "location=chicago" https://api.example.com/search 

This searches for "pizza" in the location "chicago". The API docs should specify the parameter names.

Download a File

Use -o to save the response directly to a file:

curl -o myfile.zip https://example.com/files/myfile.zip

This downloads the myfile.zip file from example.com.

Automate with Scripts

You can integrate cURL into scripts to automate sending GET requests:

#!/bin/bash

URL="https://api.example.com/data"

curl -s "$URL" > data.json

Save this bash script and run it to automatically fetch data from an API and save the response to a file.

Conclusion

cURL is a versatile command line tool that makes it easy to experiment with different types of HTTP requests. With a few simple options, you can send customizable GET requests to extract data from APIs and websites.

The key takeaways are:

• Use curl followed by a URL to send a basic GET request
• Add parameters with -G and -d
• View response headers with -i or --include
• Get JSON data by sending the Accept: application/json header
• Follow redirects using -L or --location
• Automate GET requests by wrapping curl in scripts

cURL has many more options for advanced usage like proxying requests, authenticating requests, tweaking performance, and more. Refer to the cURL man pages for detailed documentation on all available options.

I hope this guide provided you a solid foundation for sending GET requests with cURL! Let me know if you have any other questions.