A Beginner‘s Guide to Building Datasets for Machine Learning with Web Scraping

ByJordan Brown

In the world of artificial intelligence and machine learning, data is king. The quality and quantity of data you feed into your models directly impacts their performance. But gathering that data can be challenging, especially when you have niche requirements that off-the-shelf datasets don‘t cover. That‘s where web scraping comes in.

Web scraping is the process of automatically collecting data from websites using software. With web scraping, you can gather text, images, numbers, and other types of data at scale and convert it into a structured format for data science and machine learning applications.

According to a 2020 Oxylabs survey, 30.1% of data scientists and machine learning professionals use web scraping as a method to gather training data for models. And the market for web scraping tools and services is projected to reach over $10 billion by 2025 per Global Market Insights.

In this guide, we‘ll take a deep dive into the fundamentals of web scraping with Python. You‘ll learn how web scraping works under the hood, how to write scrapers to collect data from websites, best practices to keep in mind, and legal and ethical considerations. Let‘s jump in!

Understanding the Fundamentals of Web Scraping

At its core, web scraping is built on top of the request-response cycle of HTTP (Hypertext Transfer Protocol). When you visit a webpage in your browser:

  1. The browser (client) sends an HTTP request to the web server hosting the page
  2. The server processes the request and sends back an HTTP response containing the page content
  3. The browser renders the HTML content of the response to display the page

Web scraping works in a similar way, but instead of a browser, you programmatically send requests using a script or tool. The server still sends back the same response, but instead of rendering it, you parse the HTML to extract the data you‘re interested in.

Some key concepts to understand in the request-response cycle include:

  • HTTP Methods: GET (retrieve data), POST (submit data), PUT, DELETE, etc. GET requests are most common for basic scraping.
  • HTTP Headers: Key-value pairs sent with the request/response, like User-Agent, Content-Type, Authorization, etc. Setting appropriate headers is important for scraping.
  • Status Codes: Numbers that indicate the result of the request, like 200 OK, 404 Not Found, 500 Server Error, etc. Handling different status codes is crucial for robust scraping.

Let‘s look at a simple example of making an HTTP request in Python using the requests library:

import requests

url = ‘https://quotes.toscrape.com/‘
response = requests.get(url)

print(response.status_code)  # 200
print(response.headers[‘Content-Type‘])  # ‘text/html‘
print(response.text[:100])  # ‘<!DOCTYPE html><html lang="en"> ...‘

This code sends a GET request to the specified URL, and we can access various parts of the response, like the status code, headers, and content.

Finding and Extracting Data from HTML

Once you have the HTML content of a page, the next step is picking out the data you want to extract. Modern websites are complex, with deeply nested HTML structures, so finding the right elements can be tricky.

The most valuable tool for dissecting a page‘s HTML is your browser‘s built-in developer tools, usually accessible via F12 or right-click > Inspect. In the Elements tab, you can explore the HTML tree, click elements to see their properties, and test out CSS selectors and XPath expressions to isolate specific elements.

For example, on the Quotes to Scrape page, if you inspect one of the quotes, you‘ll see it‘s contained in a <div> with a class of quote:

<div class="quote">
    <span class="text">"The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking."</span>
    <span class="author">by Albert Einstein</span>
    <div class="tags">
        <a class="tag" href="/tag/change/page/1/">change</a>
        <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>
        <a class="tag" href="/tag/thinking/page/1/">thinking</a>
        <a class="tag" href="/tag/world/page/1/">world</a>
    </div>
</div>

To extract the quote text, author, and tags, we could use the following CSS selectors:

  • Quote text: .quote .text
  • Author: .quote .author
  • Tags: .quote .tag

Here‘s how we could extract this data using Python and BeautifulSoup:

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, ‘html.parser‘)

quotes = []
for quote in soup.select(‘.quote‘):
    text = quote.select_one(‘.text‘).get_text(strip=True)
    author = quote.select_one(‘.author‘).get_text(strip=True)
    tags = [tag.get_text(strip=True) for tag in quote.select(‘.tag‘)]

    quotes.append({
        ‘text‘: text,
        ‘author‘: author,
        ‘tags‘: tags
    })

This code finds all elements matching the .quote selector, then for each one, extracts the text, author, and tags, storing them in a dictionary that gets appended to the quotes list.

A Real-World Web Scraping Example

Let‘s put this all together with a more realistic web scraping example. We‘ll scrape book data from Books to Scrape, a mock online bookstore designed for practicing web scraping.

Our goal will be to collect the title, price, rating, and availability of each book from all 50 pages of the site, and store the data in a SQLite database.

Here‘s the complete code with explanations:

import requests
import sqlite3
from bs4 import BeautifulSoup

base_url = ‘http://books.toscrape.com/catalogue/page-{}.html‘

def scrape_page(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, ‘html.parser‘)

    books = []
    for book in soup.select(‘.product_pod‘):
        title = book.select_one(‘h3 a‘)[‘title‘]
        price = book.select_one(‘.price_color‘).get_text(strip=True)
        rating = book.select_one(‘p.star-rating‘)[‘class‘][1]
        availability = book.select_one(‘.availability‘).get_text(strip=True)

        books.append((title, price, rating, availability))

    return books

def main():
    conn = sqlite3.connect(‘books.db‘)
    c = conn.cursor()
    c.execute(‘CREATE TABLE IF NOT EXISTS books (title TEXT, price TEXT, rating TEXT, availability TEXT)‘)

    for page in range(1, 51):
        url = base_url.format(page)
        print(f‘Scraping {url}...‘)

        try:
            books = scrape_page(url)
        except Exception as e:
            print(f‘Error scraping {url}: {e}‘)
            continue

        c.executemany(‘INSERT INTO books VALUES (?, ?, ?, ?)‘, books)
        conn.commit()

    print(‘Done!‘)
    conn.close()

if __name__ == ‘__main__‘:
    main()

Key points:

  • We define a scrape_page function that takes a URL, extracts the book data from the page, and returns it as a list of tuples
  • In main, we connect to an SQLite database and create a books table if it doesn‘t exist
  • We generate the URL for each page in a loop, call scrape_page for each one, and insert the results into the database
  • We wrap the scrape_page call in a try/except block to catch and log any errors that occur during scraping, so one bad page doesn‘t stop the entire script
  • We print progress messages to keep track of the scraping process

This example demonstrates several best practices:

  • Separating the scraping logic into a reusable function
  • Generating URLs programmatically to handle pagination
  • Storing results in a database for easy querying and analysis
  • Handling errors gracefully to avoid crashes
  • Logging progress to monitor the scraping job

Data Quality Considerations

When preparing datasets for machine learning via web scraping, ensuring data quality is crucial. Some issues to watch out for include:

  • Inconsistent data formats (e.g. prices with/without currency symbols, dates in different formats)
  • Missing or null values
  • Outliers or incorrectly parsed values
  • Duplicate records

To catch and handle these issues, it‘s a good idea to add data validation and cleaning steps to your scraping pipeline. For example:

  • Remove whitespace and standardize formats for fields like prices, dates, etc.
  • Check for missing required fields and decide how to handle (e.g. drop rows, fill with a default)
  • Validate numeric fields are within expected ranges
  • Remove rows with malformed or invalid data
  • Drop duplicate records based on a unique identifier

Here‘s an example of adding some basic data validation to our book scraping function:

def scrape_page(url):
    # ...

    for book in soup.select(‘.product_pod‘):
        # ...

        # Validate rating is one of the expected values
        valid_ratings = [‘One‘, ‘Two‘, ‘Three‘, ‘Four‘, ‘Five‘]
        if rating not in valid_ratings:
            print(f‘Invalid rating "{rating}" for "{title}", skipping...‘)
            continue

        # Clean price
        price = float(price[1:])

        # Validate price is within expected range
        if price < 0 or price > 100:
            print(f‘Unexpected price "{price}" for "{title}", skipping...‘)
            continue

        # ...

Adding these kinds of checks to your scraper can help ensure a higher quality dataset for your machine learning pipeline.

Legal and Ethical Scraping

When scraping websites, it‘s important to consider the legal and ethical implications. Some key points to keep in mind:

  • Check the website‘s robots.txt file and respect any prohibitions on scraping
  • Review the site‘s terms of service for language related to data collection and use
  • Don‘t overload the site‘s servers with rapid-fire requests, use delays and limit concurrency
  • Consider the purpose and sensitivity of the data you‘re collecting, especially if it contains personal information
  • Use the scraped data only for its intended and allowed purposes, not to compete with or harm the site owner

Some famous court cases have set legal precedent around web scraping:

  • In Hiq Labs v. LinkedIn (2019), the US Court of Appeals ruled that scraping publicly accessible data is legal
  • In Ryanair v. PR Aviation (2015), the European Court of Justice upheld Ryanair‘s right to restrict scraping in its terms of service

Legal rulings like these showcase the importance of gaining explicit permission for scraping when required, and being mindful of jurisdiction. When in doubt, it‘s best to consult legal counsel.

From an ethical perspective, the Web Scraping Code of Conduct is a good set of guidelines to follow. It emphasizes respect for website owners and users, transparency in your scraping practices, and using scraped data responsibly.

Conclusion

Web scraping is a powerful tool for assembling datasets for machine learning, but it comes with both technical and ethical challenges. By understanding how scraping works, how to write quality scrapers, and how to operate within legal and ethical bounds, you can leverage web data to power your AI applications.

In this guide, we‘ve covered:

  • The basics of HTTP and how web scraping works
  • Analyzing page structure with browser dev tools
  • Extracting data with Python and BeautifulSoup
  • Building a complete scraping pipeline with error handling and data storage
  • Validating and cleaning web data for ML
  • Legal and ethical best practices for scraping

Of course, this only scratches the surface of what‘s possible with web scraping. To learn more, check out resources like Web Scraping with Python by Ryan Mitchell or the Beautiful Soup documentation.

Treat the web and its data with respect, and happy scraping!

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts