Scraping Alternative Data: Technological Challenges to Keep in Mind

Introduction

Alternative data has exploded onto the business scene as companies race to supplement traditional sources with insights extracted from unconventional digital tea leaves. From web traffic and social chatter to satellite images and credit card transactions, this "extra-financial" data reveals the trends and patterns needed to predict shifts in markets, assess risks, spot opportunities and guide strategy.

No wonder alternative data has become a multi-billion dollar industry seemingly overnight. A survey by Greenwich Associates found that 85% of institutional investors now leverage alternative data sets in their investment workflows. Man Group alone has budgeted $100 million to incorporate alternative data into its trading algorithms and models. The total spend on alternative data among hedge funds and investment banks topped $7 billion in 2020 according to Opimas, and will continue rocketing upwards at a 50% annual growth rate.

With the competitive advantages so clear, every company wants to tap into rich alternative data streams from across the web and beyond. However, collecting and using alternative data effectively at scale requires reckoning with some significant technological hurdles:

• Building and operating industrial-scale data harvesting pipelines.
• Parsing messy unstructured data of dizzying variety.
• Refreshing data constantly to avoid decay.
• Securing legal access to data sources.
• Ensuring compliance with regulations and ethics.
• Protecting data integrity and security.

In this guide, I‘ll share key challenges I frequently encounter when helping clients implement alternative data strategies, along with proven solutions to overcome them. With the right scaffolding in place, you can responsibly mine alternative data to enhance insights and guide smarter decisions. Let‘s dig in!

The Mounting Complexities of Scraping Alternative Data

During my 5 years in the web scraping industry advising clients across retail, investing, real estate and other sectors, I‘ve seen alternative data collection efforts encounter the following core challenges:

Sheer Data Scale and Velocity

Alternative datasets like social conversations, reviews, transaction logs and satellite imagery reach massive scales of billions of data points, totalling terabytes or petabytes. This data gushes from thousands of sources constantly. Just harvesting and storing this firehose of information demands robust, industrial-sized data pipelines.

To illustrate the scale, BrightData recently scraped ~700 million Amazon reviews. A hedge fund scraping ship tracking data from marine websites ingests over 200 GB per day. Scraping at this velocity requires high-bandwidth datacenter proxies and auto-scaling cloud infrastructure. Limits must be eliminated through large proxy pools and distributed scraping.

High Dimensionality and Messy Data

While traditional financial data arrives neatly structured, alternative data is largely unstructured text, images, video, audio and more. This extreme variety means scrapers deal with HTML, JSON, feeds, APIs and databases simultaneously. Vast dimensionality makes parsing the signal from the noise exceedingly difficult.

Advanced techniques like NLP, computer vision, OCR, geospatial analysis and sentiment analysis help make sense of all this messy data. The data team at a $40 billion AUM asset manager told me they still spend over 80% of their time just "cleaning and organizing" scraped alternative data before deriving insights.

Constantly Shifting Data Landscapes

Unlike quarterly filings, alternative data sources never stop. Public sentiment, product demand, corporate initiatives and other trends reflected in this data evolve hour by hour. Data swiftly becomes stale if not refreshed via perpetual scraping.

Strategically scheduling and orchestrating scrapers is imperative to keep alternative data pipelines flowing with fresh info. Leading hedge funds re-scrape key sources of alternative data every hour or day rather than weekly or monthly. They combine incremental scraping with intelligent data warehousing to manage the velocity.

Walled Garden Data Silos

While data on public websites can be freely scraped, many high-value alternative data wells like social platforms, commercial data sets, and company intranets are locked behind paywalls, credentials, and strict terms of service. Legal access must be secured.

Securing subscriptions, partnerships, or APIs can unlock proprietary data streams. For public sources, robust proxies that mimic real human visitors can help avoid blocks. However, these approaches cost time and money. The head of data at a top quantitative fund told me over 35% of their alternative data access costs go towards these kinds of gate-keeping fees.

Increasing Compliance Burden

With alternative data touching personal lives and powering mission-critical business decisions, the regulatory stakes around proper data use have risen. Strict new data privacy laws like GDPR and CCPA have emerged. Ethics around "overscraping" websites and respecting data rights are equally important.

Navigating compliance when scraping alternative data demands continuous review of terms of service, consultation with legal counsel, and adjustment of practices to honor user consent and rights. My clients are wise to address compliance upfront rather than playing catch-up.

Escalating Data Security Risk

The distributed nature of scraping disparate third-party sources in the cloud expands the attack surface area for alternative data systems. Meanwhile, the sensitivity of some alternative data makes it a prime target for malicious actors or unethical competitors.

A vice president at a large hedge fund revealed to me that since they began alternative data collection, cyberattacks on their systems have tripled. With so much at stake, rigorous controls around data access, encryption, perimeter security, and workforce training are required.

Overcoming the Hurdles Through Responsible Scraping

Facing this mounting complexity, how can organizations successfully tap alternative data without being overwhelmed? Here are smart solutions I‘ve guided clients through to address each core challenge:

Handle High Volumes:

• Leverage tens of millions of residential and datacenter proxies via vendors like BrightData, Oxylabs and Soax to distribute scrapes.
• Build auto-scaling scraper fleets on cloud platforms like AWS Lambda, Google Cloud Run, or Azure Virtual Machine Scale Sets.
• Containerize scrapers via Docker and Kubernetes for easy replication across machines.
• Use robust middleware platforms like Scrapy Cloud, Portia, and ParseHub to manage and scale distributed scraping.

Clean Up Messy Data:

• Refine scrapers with tools like Google Cloud Vision for image extraction, AWS Comprehend for NLP, MonkeyLearn for classification, and Phantombuster for web data APIs.
• Post-process scraped data with ETL tools like Seal, Spark, and Trifacta to filter, normalize, and structure it.
• Deduplicate records with deterministic, fuzzy matching across keys like names, addresses, etc.
• Establish data oversight team charged with raising data quality standards.

Keep Data Fresh:

• Schedule regular scraping via cron jobs, business process automation platforms, or workflows tools like Airflow.
• Incrementally update databases/warehouses with new scraped data via chunked inserts, upserts etc.
• Automatically trigger rescrapes when certain database metrics (size, staleness) are exceeded.
• Build feedback loops to identify when source data has changed and scraping needs re-running.

Access Walled Data:

• Evaluate APIs from the start as a sanctioned means of large-scale data access if available.
• Forge win-win data sharing partnerships withproviders; pool budgets if needed.
• Consult legal counsel to ensure ToS compliance when scraping behind login walls.
• Route traffic through fresh proxy pools to mimic new users and avoid blocks.

Observe Compliance:

• Maintain sharp, up-to-date understanding of relevant data laws in all geographies touched by your data supply chain.
• Continuously monitor sources‘ ToS and robots.txt directives to adapt scraping accordingly.
• Anonymize collected data wherever possible to safeguard identities.
• Build required consent flows for data collection and retention policies into scraper workflows.

Lock Down Data:

• Classify data sensitivity and establish least-privilege access guided by need.
• Encrypt scraped data in transit and at rest; securely manage keys.
• Provision separate environments for dev vs. prod data to limit access.
• Pen-test systems repeatedly; fix vulnerabilities before scaling up data collection.
• Install rigorous cybersecurity protocols for cloud services, hardware, networks, and personnel.

Scraping alternative data certainly poses thorny technological dilemmas. But by proactively addressing these challenges, you can avoid missteps and focus instead on extracting game-changing insights from alternative data. Reach out anytime if you need help building a sound data strategy tailored to your specific business objectives and pain points! I‘m always happy to chat.

Conclusion: Scraping the Competitive Edge

In closing, alternative data represents a paradigm shift in how organizations understand themselves, their customers, and their markets. While unlocking its full value requires reckoning with issues of scale, complexity, legality and security, solutions exist.

With careful strategy, robust infrastructure, and ethical practices, companies can mine alternative data successfully to enhance critical business decisions. But reckless and naive scraping can risk legal exposure, faulty insights, and technical crashes. Scraping alternative data responsibly – neither ignoring challenges nor being daunted by them – is key to competitive advantage.