The Ultimate Beginner‘s Guide to Web Scraping in R with rvest: Hands-On Tutorial
Introduction
Web scraping is the process of extracting data from websites. It‘s a valuable skill for any data scientist or analyst to have in their toolkit. Even if the data you need isn‘t available in a convenient format like CSV or via an API, you can often obtain it by scraping it directly from a webpage.
The internet contains a vast trove of information that can be extremely useful for data science projects. By scraping data from the web, you can access everything from text to images to statistical data. Some examples of data you might want to scrape include:
- Product details and prices from e-commerce sites
- User reviews and feedback
- Sports scores and statistics
- News headlines and article text
- Social media posts
- Movie and book details, ratings, etc.
Web scraping automates the tedious process of manually copying and pasting data from websites. With a little bit of coding, you can extract large amounts of data very quickly and efficiently.
In this tutorial, I‘ll walk through how to perform web scraping using R and the rvest package. By the end, you‘ll be able to scrape data from any webpage and be well on your way to harnessing the full power of data on the internet! Let‘s get started.
How Web Scraping Works
At a high level, web scraping involves programmatically downloading the HTML source code of a webpage and extracting the data you want from it. HTML is the language that webpages are written in – it defines the structure and content of the page.
For example, here is a simplified snippet of HTML:
<html>
<body>
<h1>Welcome!</h1>
<p>Here is a paragraph.</p>
<ul>
<li>List item 1</li>
<li>List item 2</li>
</ul>
</body>
</html>
This HTML contains a top level heading, a paragraph of text, and an unordered list with two items. When rendered in a web browser, it would look something like:
Welcome!
Here is a paragraph.
- List item 1
- List item 2
To scrape this data, we would need to download this HTML and then parse out the pieces we want, such as the text of the heading and list items.
Websites also use CSS to define the visual style and layout of the page. CSS selectors are a way to specify an element or group of elements on the page. For example, the CSS selector "h1" would select the
heading element, while "li" would select all
More specific CSS selectors allow targeting elements based on their class, ID, position in the document, and more. Learning to come up with the right selectors to precisely extract the elements you want is a key part of web scraping.
Setting Up
To get started, you‘ll need:
- R installed
- The rvest package installed
- A basic familiarity with R and how to run code
You can install rvest with:
install.packages("rvest")We‘ll also be using the SelectorGadget browser extension to make it easier to find the right CSS selectors for elements we want to extract. Install it from here:
Once installed, the extension will appear as a box with a crosshairs icon in your browser toolbar. Activating it allows you to click on elements in the webpage and see the corresponding CSS selector highlighted.
Scraping a Simple Table
Let‘s start with a simple example to get our feet wet. We‘ll scrape the list of USA population by state from Wikipedia:
https://en.wikipedia.org/wiki/List_of_U.S._states_and_territories_by_population
First, use SelectorGadget to find the CSS selector for the main table on the page. Click the element, then SelectorGadget will highlight what it thinks the selector should be in green. Adjust the selection with the + and – tools until just the table is highlighted.
In this case, the selector that captures the table is:
".wikitable"
Now we‘re ready to scrape the table with rvest! Here‘s the code:
library(rvest)
url <- "https://en.wikipedia.org/wiki/List_of_U.S._states_and_territories_by_population"
population_table <- url %>%
read_html() %>%
html_nodes(".wikitable") %>%
html_table()
print(population_table)Let‘s break this down:
- The url variable contains the URL of the page we want to scrape
- read_html() downloads the HTML of the page
- html_nodes(".wikitable") selects all elements matching our ".wikitable" CSS selector
- html_table() parses the table elements and reads the data into a data frame
Run this code and you should see the population data printed out!
We can make a couple enhancements to clean up the data:
population_table <- url %>%
read_html() %>%
html_nodes(".wikitable") %>%
html_table(header = TRUE) %>%
.[[1]]
colnames(population_table) <- make.names(colnames(population_table))
print(head(population_table))Setting header = TRUE ensures the first row is interpreted as column headers. .[[1]] extracts the first (and only) matched table. make.names() cleans up the column names to be syntactically valid.
We now have a clean data frame with columns for Rank, State, Population, Census, and more. You could easily export this to CSV or feed it into further analysis in R.
A More Complex Example
Let‘s tackle a more complex example that involves paginated data spread across multiple pages. We‘ll scrape player stats for the top 100 home run hitters in MLB history from Baseball Reference:
https://www.baseball-reference.com/leaders/HR_career.shtml
The data we want is in a table, but it‘s split across multiple pages, 50 rows at a time. We‘ll need to iterate through each page and combine the results.
Using SelectorGadget, we can see that the player data table has a selector of ".stats_table", while the "Next Page" link can be selected with ".button2:nth-child(1)".
Here‘s the full code to collect all the data:
library(rvest)
url <- "https://www.baseball-reference.com/leaders/HR_career.shtml"
extract_data <- function(html) {
table <- html %>%
html_nodes(".stats_table") %>%
html_table(header = TRUE) %>%
.[[1]]
next_link <- html %>%
html_nodes(".button2:nth-child(1)") %>%
html_attr("href")
return(list(table=table, next_link=next_link))
}
scrape_page <- function(url) {
results <- url %>% read_html() %>% extract_data()
table <- results$table
next_link <- results$next_link
if (length(next_link) > 0) {
next_url <- paste0("https://www.baseball-reference.com", next_link)
table <- rbind(table, scrape_page(next_url))
}
return(table)
}
hr_data <- scrape_page(url)
print(head(hr_data))The key steps are:
- extract_data() takes a page‘s HTML and extracts the table and URL for the next page
- scrape_page() reads the HTML for a given URL, extracts the table, checks if there is a next page, and if so recursively calls itself on the next page‘s URL
- rbind() combines the data from each page into one data frame as we iterate through the pagination
This recursively traverses through all the pages until there is no "next page" link found, collecting the data into a single data frame.
With this approach, we are able to scrape the top 100 home run hitters, getting their rank, name, HR total, and other stats. The code can be adapted to paginated data in other formats as well.
Cleaning and Outputting Data
Raw HTML data often needs cleaning and standardizing before it‘s ready for analysis. Some common issues to handle:
Missing data: not all fields may be populated for each row. You may need to filter out rows with missing key fields or decide on a strategy for handling missing values.
Inconsistent formatting: numbers, dates, etc. may be in different formats that need to be standardized
Combining data: if scraping data from multiple pages or sections, you‘ll need to combine it sensibly into one unified data frame
Luckily, R has many great tools for data cleanup. Some key functions:
- is.na() checks for missing values
- parse_number() converts string numbers to numeric
- as.Date() converts strings to dates
- unite() and separate() for combining and splitting columns
- filter() and select() for subsetting data
- regular expressions for extracting data from unstructured text
Once your data is cleaned, you‘ll likely want to save it to disk so you have it on hand for analysis. write.csv() allows saving a data frame to CSV format:
write.csv(hr_data, "top_100_hr_hitters.csv", row.names=FALSE)Ethics and Best Practices
There are some important considerations to keep in mind when scraping data from websites:
- Check for terms of service and robots.txt
Many websites have terms of service or a robots.txt file that specify if and how they want their data to be accessed. You should always respect what the website owner intends. Some may disallow scraping completely.
- Don‘t overload the website with requests
Scraping can put a burden on the web servers you are collecting data from. Make sure to space out your requests so you aren‘t hitting the server too rapidly. You may need to add pauses or rate limiting to your code.
- Identify yourself
It‘s a good practice to identify your scraper to website owners by including a user agent string with contact info and an explanation of what you are doing. That way they can contact you if needed.
- Use data responsibly
Be ethical with how you use data collected from the web. Don‘t expose people‘s personal info or use data to manipulate markets or exploit others. Give attribution to sources where appropriate.
- Understand the legality
In some jurisdictions, there are legal restrictions on how you can use data scraped from websites. Copyright and data privacy laws may apply. It‘s a complex topic, so do research on what rules apply to your specific use case.
Next Steps
Congratulations, you now have the skills to scrape a wide variety of data from the web using R! Some next steps you can explore:
- Scrapy is a popular Python framework for scraping that makes it easy to crawl entire websites
- NLP libraries like quanteda allow doing interesting text analysis on scraped data
- Selenium can be used to scrape dynamic pages that load data via JavaScript
- Computer vision techniques can extract data from images
The possibilities are endless. I wish you the best of luck on your web scraping adventures! Leave a comment with the most interesting data you‘ve scraped.
