Extracting URLs from Websites: The Definitive Guide

My 10+ Years of Experience Using Selenium and Other Tools

As an expert proxy provider and web data extraction specialist with over a decade of hands-on experience, I‘ve mastered the intricacies of discovering and capturing URLs at scale across even the most complex and dynamic websites.

Through hundreds of client projects, I‘ve honed an analytical approach for assessing site structures and developing tailored scraping solutions. I‘d like to share some of that extensive knowledge with you today.

By the end of this comprehensive 2,500+ word guide, you‘ll be equipped with:

• A wide range of techniques and tools for uncovering hidden links
• Code snippets and examples grounded in real-world applications
• My insider tips for overcoming common extraction obstacles
• Frank advice on proxy services I‘d genuinely recommend
• And so much more!

I aim to educate but also simplify what can often be an opaque, technical landscape. My goal is for you to walk away with clarity and confidence to start implementing these URL extraction strategies yourself.

So without further ado, let‘s dive right in! This will be the only resource you need when determining how to find all URLs on pretty much any site thrown your way…

An Introduction to URL Extractors

Before jumping into the popular Selenium library, I wanted to provide quick overviews of some other common URL extraction tools:

Regular Expressions

While limited for complex sites, regex can be used to find URL patterns in simple HTML or text content. For example:

import re

text = "For details visit https://www.example.com and http://www.examples.org"

pattern = r"https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&//=]*)"

urls = re.findall(pattern, text)
print(urls)

BeautifulSoup

BeautifulSoup is a Python library for parsing HTML and XML content. It allows you to traverse the document tree and extract tags like anchors:

from bs4 import BeautifulSoup
import requests

content = requests.get("https://example.com").text 
soup = BeautifulSoup(content, "html.parser")

for anchor in soup.findAll("a"):
   print(anchor["href"])

Scrapy Framework

Scrapy is a dedicated web crawling and scraping framework. It boasts advanced extraction capabilities but requires more code investment:

import scrapy

class UrlSpider(scrapy.Spider):
    name = "urlcrawler"     
    start_urls = ["http://example.com"]

    def parse(self, response):
        for anchor in response.xpath(‘//a‘):
            yield {"url": anchor.xpath("@href").get()} 

Alright, now that we‘ve covered the basics of other tools, let‘s focus on our star attraction – Selenium!

Getting Started with URL Extraction Using Selenium

Selenium is likely the most popular and robust browser automation suite available today. I generally recommend Selenium for scraping complex, heavy JavaScript sites where content loads dynamically.

Here‘s a basic script outline:

from selenium import webdriver

browser = webdriver.Chrome()
browser.get("https://example.com")   

# Find/extract URLs

browser.quit()

Next we need to locate the key page elements housing our target URLs.

Selenium provides a number of built-in discovery methods, mainly:

browser.find_element_by_id() 
browser.find_element_by_name()
browser.find_element_by_xpath()
browser.find_element_by_link_text()
browser.find_element_by_partial_link_text()  
browser.find_element_by_tag_name()
browser.find_element_by_class_name()
browser.find_element_by_css_selector() 

For URLs, we usually want the href attributes from anchor tags. Let‘s grab those:

anchors = browser.find_elements_by_tag_name("a")

for anchor in anchors:
   print(anchor.get_attribute("href"))  

And there‘s our URL list! Pretty straightforward, but definitely room for improvement…

Expert Techniques for Finding All URLs

Through extensive trial-and-error on hundreds of sites, I‘ve compiled a robust set of approaches that I apply sequentially to uncover all links.

Let me share some of my top pro-tips:

Handle Duplicate Values

Using Sets eliminates duplicates while maintaining order:

urls = set()

for anchor in anchors:
   urls.add(anchor.get_attribute("href"))

print(list(urls)) 

Scroll Entire Page

Hidden links can load as you scroll down:

browser.execute_script("window.scrollTo(0, document.body.scrollHeight);") 

Click Hidden Expanders

Interact with elements that display more content:

buttons = browser.find_elements_by_css_selector(".expander")

for button in buttons:
   button.click()

Wait for Elements

Let page fully load before extracting:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(browser, 20)

anchors = wait.until(EC.visibility_of_all_elements_located((By.TAG_NAME, "a"))) 

Regex Pattern Matching

Isolate links by patterns they match:

import re
pattern = r"/archive/"

for anchor in anchors:
   url = anchor.get_attribute("href") 
   if re.search(pattern, url):
      print(url)

And those are just a few examples! When in doubt, lean on these proven methods above all else.

Now let‘s walk through some detailed real-world cases.

Case Studies: Extracting URLs from Actual Sites

Up until now, we‘ve explored the how of link extraction using Selenium. Next I want to provide concrete examples demonstrating the what and why through several case studies from previous client engagements.

These are lightweight examples – in practice my scripts often exceed 500+ lines across multiple classes and files. But the core principles remain the same.

Let‘s take a look:

Case 1: Directory Site

One common client request is aggregating links found within website directories.

Let‘s try a script on Delicious Brains, a popular WordPress resource site.

Goal: Extract all links listed on main resources page

Approach:

• Grab all anchors
• Eliminate internal sitenavigation
• Remove external site formatting
• Output to JSON

Script:

import json
import re  

browser.get("https://deliciousbrains.com/resources/")

internal = re.compile(r"^/(?!wp-content)") 

anchors = browser.find_elements_by_tag_name("a")

cleaned_urls = []

for anchor in anchors:
   url = anchor.get_attribute("href")

   if re.search(internal, url): 
      continue

   url = re.sub(r"/[^/]+/?$", "", url)
   cleaned_urls.append(url)

with open("db-resources.json", "w") as f: 
   json.dump(cleaned_urls, f)

This locates all links on the page, filters out internal site navigation URLs, removes extraneous dependencies from external sites, and formats the final list cleanly into a .json file.

Case 2: Pagination Site

Another request is scraping websites with pagination where URLs span multiple pages.

For example, Internet Archive‘s Prelinger collection has hundreds of great public domain videos across dozens of pages.

Let‘s grab all the collection URLs:

Goal: Extract all video URLs across pagination

Approach:

• Click through pagination
• Grab links
• Follow pattern
• Eliminate duplicates
• Output to text file

Script:

import re
import time

urls = set() 

browser.get("https://archive.org/details/prelinger&tab=collection")  

while True:

    anchors = browser.find_elements_by_css_selector(‘div.item-ttl C‘)

    for anchor in anchors:

        url = anchor.find_element_by_tag_name("a").get_attribute("href")

        if re.search(r"details/.+", url):  
            urls.add(url)            

    next = browser.find_elements_by_xpath(‘//a[@title="Next Page"]‘)

    if not next:
        break

    next[0].click()

    time.sleep(5)

with open("archive-vids.txt", "w") as f:
   f.write("\n".join(list(urls)))

This clicks through all video pages, grabs individual URLs, eliminates duplicates, and outputs everything into an easy text list.

As you can see, while the goals vary, the core Selenium principles remain consistent across projects.

Case 3: JavaScript Site

For this next one, let‘s tackle a site loaded entirely through JavaScript.

Pages like FlipHTML5 can seem daunting, but my script below handles it with precision:

Goal: Extract all book links from digital book viewer

Approach:

• Wait for JavaScript elements
• Scroll through books
• Allow load delays
• Capture based on selector

Script:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

browser.get("https://fliphtml5.com/web-book-demo/")  

delay = 10

last_height = browser.execute_script("return document.body.scrollHeight")  

while True:

    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")  

    time.sleep(delay) 

    new_height = browser.execute_script("return document.body.scrollHeight")


    if new_height == last_height:

        break

    last_height = new_height

wait = WebDriverWait(browser, 10)

books = wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME , "demoBook")))

for book in books:

    url = book.find_element_by_css_selector(".demoBookImg").get_attribute("href") 
    print(url)

By leveraging waits, delays, and JavaScript execution – we can scrape even the most complex single page applications with ease!

Common URL Extraction Pitfalls

Through extensive troubleshooting of problematic sites, I‘ve compiled solutions for several recurrent link extraction issues:

Dynamic URLs

URLs can generate uniquely on each visit or session making links difficult to validate.

Solution: Analyze pattern in network tool, extract base then append generated component

Pagination Skipping

Some paginated sites do not have working "Next" buttons or page numbers.

Solution: Scroll down to trigger next page, detect changes in elements to confirm

Bot Protection

Aggressive sites may block Selenium and other automation tools entirely.

Solution: Use proxy rotation, headless browser, or browser automation detection evasion techniques

Runaway Memory Usage

Creating new browsers constantly instead of reusing can balloon memory footprint.

Solution: Close browser instances appropriately, implement context managers to handle opening/closing automatically

I have an extensive repository of such solutions for virtually any scrape blocking obstacle. Reach out if one has you temporarily stuck!

Recommended Proxies for Smooth Scraping

As an experienced proxy provider catering to Fortune 500 companies, I have worked closely with all major residential proxy services.

If you do intend to scrape at scale, proxies are absolutely vital for masking scrapers and avoiding blocks.

Here are my top recommendations based on proven performance and reliability:

Smartproxy

Smartproxy offers one of the largest and fastest proxy networks with over 10 million IPv4 addresses spanning 130+ countries. Support and infrastructure are top-notch.

Soax

Soax features excellent transparent proxy support for Selenium with sustainable IP refresh rates. Recently expanded infrastructure to 2 million elite residential IPs.

GeoSurf

While smaller, GeoSurf pioneered the residential proxy market a decade ago. Tight-knit team with specialized scraping consultation.

I can personally vouch for each based on hundreds of successful client integrations. Their products offer the perfect blend of quality, sophistication, and transparency needed for smooth site data extraction.

Let‘s Recap Those Key Lessons

If you‘ve made it this far, congratulations! I know that was an overwhelming amount of information but also hopefully tremendously valuable.

Let‘s recap the core concepts for finding and extracting URLs with Selenium:

• Master fundamental element location and selection techniques
• Apply sequential unraveling tactics like scrolling, clicking, and waiting
• Employ advanced filters like regex to isolate target URL patterns
• Optimize performance by eliminating duplicates and closing browsers
• Implement proxies and other evasion tactics to avoid blocks
• Know common pitfalls and corresponding troubleshooting techniques

Those foundational skills will enable you to extract links efficiently across the vast majority of sites.

For additional help or project-based scraping consultation, feel free to reach out!

Now go forth and harvest those URLs with confidence!