Democratizing Machine Learning with Google BigQuery ML and SQL

The field of machine learning has exploded over the past decade, with breakthroughs in deep learning enabling incredible feats like near-human image classification, natural language understanding, and even defeating world champions at complex strategy games. However, much of this progress has been limited to the rarified world of academic research labs and the engineering departments of giant tech companies.

For machine learning to truly transform industries and deliver widespread impact, it needs to become more accessible to the millions of business analysts, BI practitioners, and SQL developers working with data every day. These knowledge workers deeply understand their domains and have keen insight into the problems machine learning could help solve, if only they had the tools and skills to apply it.

Enter BigQuery ML, a groundbreaking service from Google Cloud that enables users to build and deploy state-of-the-art machine learning models using only simple SQL queries. By abstracting away the complexities of data science frameworks and exposing the power of ML through the familiar SQL syntax, BigQuery ML represents a major step forward in the democratization of machine learning.

How BigQuery ML Works

At a high level, BigQuery ML allows users to create and train machine learning models directly inside BigQuery, Google Cloud‘s massively scalable enterprise data warehouse. The core idea is to bring ML to where the data already lives and put it in the hands of SQL users.

Under the hood, BigQuery ML leverages Google‘s battle-tested TensorFlow framework to train and serve the models. When a user issues a SQL query to create a model, BigQuery translates it into a TensorFlow computation graph and executes the training across a cluster of machines. The resulting model artifacts are stored back in BigQuery tables for easy use in prediction and analysis workflows.

BigQuery ML supports supervised learning models for both regression (linear regression) and classification (logistic regression, deep neural networks), as well as unsupervised models like k-means clustering and matrix factorization. It also provides a suite of functions for evaluating model performance, feature preprocessing, and hyperparameter tuning.

One of the key innovations in BigQuery ML is its use of SQL as the interface for machine learning. With a few intuitive extensions to standard SQL, users can express complex model architectures and configurations. For example, here is the SQL statement to create a deep neural network classifier in BigQuery ML:

CREATE OR REPLACE MODEL `mydataset.dnn_model`
OPTIONS(model_type=‘dnn_classifier‘, 
        hidden_units=[64, 32], 
        activation_fn=‘relu‘,
        batch_size=64, 
        learning_rate=0.01) AS
SELECT
  label,
  feature1,
  feature2, 
  ...
FROM `mydataset.input_table`

This single SQL statement encapsulates a multi-layer neural network architecture with 64 and 32 hidden units, ReLU activations, mini-batch stochastic gradient descent training with a batch size of 64 and learning rate of 0.01. In the background, BigQuery takes care of distributing this training across many machines for speed and automatically tunes aspects like the learning rate decay schedule.

BigQuery ML in Practice

Forward-thinking companies across industries are already leveraging BigQuery ML to power advanced analytics and achieve impressive results.

Retailers are using it to build propensity models that predict which customers are likely to churn or respond to promotions. One online fashion retailer used BigQuery ML to build a classifier that identifies high-value customers likely to make repeat purchases, allowing them to target this segment with personalized offers. By operationalizing this model on their customer base of over 10 million users, they were able to improve retention by 15% and drive a 20% increase in revenue from repeat buyers.

In financial services, BigQuery ML is being used for fraud detection, credit risk assessment, and customer segmentation. A leading payment processor built a deep learning model in BigQuery ML to identify fraudulent transactions in real-time. By training on a massive dataset of over 1 billion historical transactions and hundreds of features, they were able to reduce fraud losses by 30% while keeping false positives below 0.1%.

Telcos are applying BigQuery ML to predict network outages, optimize resource allocation, and personalize subscriber plans. One major carrier used BigQuery ML to build a multi-class classifier that predicts the cause of customer support calls based on network telemetry and device diagnostics data. By proactively resolving issues before customers even notice, they lowered call volume by 20% and boosted net promoter score by 10 points.

The Business Impact

Beyond the individual use cases and success stories, BigQuery ML has the potential to fundamentally transform how businesses approach machine learning and analytics.

Traditionally, applying ML meant hiring expensive data science talent, investing in specialized infrastructure, and dealing with convoluted workflows to move and prep data. With BigQuery ML, existing data and analytics teams can get up and running with machine learning quickly using tools and languages they already know.

This drastically lowers the barriers to entry and time to value. A recent Forrester Research study found that BigQuery ML requires 43% less effort for data prep and modeling compared to traditional ML approaches. The study also estimated a 751% ROI over 3 years for a composite organization adopting BigQuery ML.

By empowering ‘citizen data scientists‘, BigQuery ML also helps address the severe global shortage of machine learning talent. The average salary for a data scientist in the US is $122,000 and there are over 300,000 unfilled positions according to LinkedIn. BigQuery ML can help fill this gap by enabling existing SQL experts to take on machine learning workloads.

Placing BigQuery ML in Context

BigQuery ML is part of a broader trend toward the democratization and consumerization of machine learning. Recognizing the strategic importance of AI and the talent scarcity, all the major cloud providers are racing to offer higher-level ML services requiring less expertise.

Amazon SageMaker, Azure Machine Learning, and Google‘s Cloud AutoML are all managed platforms that automate much of the model building and deployment workflow. Startups like DataRobot, H2O.ai, and Domino Data Lab are also gaining traction with ‘AutoML‘ platforms that apply meta-learning techniques to automatically discover optimal models and hyperparameters.

Where BigQuery ML stands out is in its seamless integration with the core data warehouse and its embrace of SQL as the lingua franca. For the legions of data analysts already using SQL to query BigQuery, building a model is now as easy as adding a few lines to their existing queries. No context switching to notebooks or Python required.

This tight coupling of data and ML is BigQuery ML‘s key differentiator. By eliminating data movement and latency, BigQuery ML enables analytics and data science teams to iterate faster and put models into production quicker. Models can be built right where the data lives and outputs can flow directly into downstream BI and reporting tools.

BigQuery ML also benefits from the core scalability and performance of BigQuery. Training complex models on billion-row datasets is simply not tractable with traditional desktop tools like R and Scikit-learn. Even popular distributed frameworks like Spark struggle with the petabyte scale of modern data warehouses. BigQuery‘s unique architecture allows it to crunch through enormous datasets quickly and scale to thousands of users.

The Road Ahead

BigQuery ML is still a young product with plenty of room for growth and improvement. Currently it supports a solid set of the most popular supervised and unsupervised learning algorithms, but the range of model types pales in comparison to mature libraries like TensorFlow and Scikit-learn.

To achieve parity with traditional ML platforms, BigQuery ML will need to expand to areas like computer vision, natural language processing, and time series forecasting. It could also go further in automating key workflows around model monitoring, retraining, and explanation.

Performance and cost are other areas to keep an eye on. While BigQuery ML trains models significantly faster than standard open-source tools, it can still be outperformed by highly optimized implementations running on premium GPU hardware. Google will need to continue innovating on both the software and hardware fronts to maintain leadership.

The full potential of BigQuery ML will also only be realized through a robust ecosystem and community. Here Google has a strong advantage with established tools like Data Studio, Looker, and Jupyter notebooks that interoperate seamlessly. But continued investment in education, documentation, and evangelism will be critical to drive adoption.

Despite these challenges, the future of BigQuery ML and the democratization of machine learning looks bright. By putting the power of ML into the hands of millions of data-fluent SQL users, BigQuery ML has the potential to accelerate the ‘consumerization‘ of AI and fundamentally change how businesses leverage their data assets.

As Gartner puts it, "By 2025, 70% of new applications developed by organizations will use low-code or no-code technologies." BigQuery ML is well positioned to ride this wave and become the default choice for enterprise ML, much like BigQuery already is for enterprise data warehousing.

In the age of AI, data is the new oil and SQL is the new shovels. BigQuery ML represents the perfect union of these two forces. The journey to democratize machine learning is far from over, but BigQuery ML is a major step in the right direction. Exciting times ahead!