Automating Machine Learning with EvalML: An Expert Guide

The field of machine learning has transformed rapidly in recent years. As the volume and variety of data continues to grow exponentially, organizations are under immense pressure to extract actionable insights at breakneck speed. At the same time, the demand for skilled data scientists far outpaces supply, with the U.S. alone facing a shortage of 250,000 data science professionals by 2024 [^1].

To help bridge this gap, a new paradigm known as Automated Machine Learning (AutoML) has emerged. AutoML is an approach that aims to automate the end-to-end process of building and deploying machine learning models, from data pre-processing to model selection, feature engineering, hyperparameter tuning, and deployment [^2]. By leveraging intelligent algorithms and vast computational power, AutoML enables data science teams to rapidly iterate and find optimal models with minimal human intervention.

The popularity and impact of AutoML is evident in its widespread adoption across industry and academia:

• 61% of enterprises plan to use AutoML in some capacity by 2024, up from 43% in 2020 [^3]
• The global AutoML market is expected to grow from $269M in 2019 to $14B by 2030, a CAGR of 45% [^4]
• Applying AutoML has been shown to increase data scientist productivity by 10X while reducing model deployment time by 90% [^5]
• Google‘s AutoML Vision service achieved an 82% accuracy rate on the ImageNet benchmark, just shy of the 84% reached by human experts [^6]

Introducing EvalML

One of the most promising open source entrants in the AutoML arena is EvalML. Developed by Alteryx, EvalML is a powerful Python library that enables data scientists to automatically build, optimize, and evaluate machine learning pipelines for supervised learning problems. With support for binary/multi-class classification, regression, and time series forecasting, EvalML is designed to accelerate the model development lifecycle and help scale data science across the enterprise.

Under the hood, EvalML leverages several optimization techniques to intelligently navigate the search space of possible pipelines:

• Bayesian Optimization: Bayesian optimization is a global optimization algorithm that builds a probabilistic model of the objective function (e.g. AUC score) and uses it to sample hyperparameter configurations that are more likely to improve performance [^7]. EvalML utilizes Bayesian optimization in its iterative pipeline search process.

• Genetic Programming: Genetic programming is an evolutionary approach that mimics natural selection to find optimal ML pipelines. It begins with an initial population of randomly generated pipelines, evaluates their "fitness", and then evolves the top performing pipelines over successive generations through mutation and crossover operations [^8]. EvalML uses genetic programming to optimize the structure and composition of pipelines.

• Early Stopping Algorithms: To save computational resources, EvalML implements early stopping logic that halts the training of ill-performing pipelines. Early stopping is triggered when the pipeline‘s performance on a validation set stagnates or degrades and has been shown to help avoid overfitting [^9].

In essence, EvalML abstracts away much of the complexities and menial work involved in machine learning, allowing data scientists to focus on framing the right problems and interpreting results. Through a clean and intuitive API, users can configure an AutoMLSearch object for a specific task, train and tune pipelines, and evaluate model performance with just a few lines of code:

from evalml.automl import AutoMLSearch

# Initialize AutoML search for binary classification
automl = AutoMLSearch(X_train=X_train, y_train=y_train, 
                      problem_type=‘binary‘, objective=‘f1‘, 
                      max_batches=25, optimize_thresholds=True)

# Find best pipeline                      
automl.search()

# Evaluate best pipeline on test set
best_pipeline = automl.best_pipeline
test_scores = best_pipeline.score(X_test, y_test)
print(test_scores)

While EvalML aims to be as hands-off as possible, it still provides plenty of flexibility for experts to customize the AutoML process to their needs. This includes the ability to:

• Specify the objective function to optimize (auc, f1, r2, etc.) or define a custom metric
• Select the pipelines and model types to consider (e.g. only linear models, no text features, etc.)
• Adjust the degree of CPU/GPU parallelism and control model training time
• Tune the internal CV folds, validation method, and threshold optimization settings
• Extract and inspect trained pipeline components, visualizations, and metadata

EvalML vs. The Competition

So how does EvalML stack up against other AutoML offerings? To help data science leaders make informed decisions, let‘s compare EvalML to some of the leading open source and commercial tools across important dimensions [^10].

Library / Product	License	ML Tasks	Pipeline Search	Compute	Interpretability
EvalML	OSS	CLS, RGR, TS	Bayesian + GP	CPU/GPU	SHAP + Charts
Auto-Sklearn	OSS	CLS, RGR	Bayesian + SMAC	CPU	PDPs + ICE Plots
TPOT	OSS	CLS, RGR	GP	CPU	Export code
H2O AutoML	OSS/Com	CLS, RGR, TS, US	Random + Stacked Ensembles	CPU/GPU	LIME, SHAP, PDP, etc.
DataRobot AutoML	Com	CLS, RGR, TS, TXT	Proprietary	CPU/GPU	Full Prediction Explanations

*Abbreviations: CLS=Classification, RGR=Regression, TS=Time Series, GP=Genetic Programming, US=Unsupervised

While not as feature-rich as some commercial platforms like H2O and DataRobot, EvalML holds its own as one of the most versatile and performant open source AutoML libraries. Its key differentiators include:

1. Broad supervised learning coverage across binary/multi-class classification, classical regression, and advanced time series problems
2. Unique combination of Bayesian optimization and genetic programming for efficient pipeline search
3. Ability to automatically tune decision thresholds for binary classification without requiring a separate step
4. Smart default handling of data types – e.g. applying NLP to text features and encoding high cardinality categorical variables
5. First class model interpretability and interactive visualization capabilities

That said, EvalML is still a relatively new library and has some room for improvement:

• Lack of distributed training and limited GPU support constrains scalability
• No explicit handling of multi-modal unstructured data like images, audio, etc.
• Comparatively smaller user community and knowledge base vs. scikit-learn based tools
• Steeper learning curve to leverage advanced customization options

Ultimately, the choice of AutoML framework depends on a team‘s specific needs, in-house skills, and existing ML infrastructure. For organizations seeking an open source solution that is highly performant, user-friendly, and versatile, EvalML is certainly a compelling option.

Expert Tips for Using EvalML

For data science practitioners looking to get the most out of EvalML, here are some technical considerations and best practices to keep in mind:

• Handling Class Imbalance: EvalML attempts to detect imbalanced classes and enables stratified splitting by default. However, in extreme cases, techniques like oversampling rare classes, applying class weights, and optimizing threshold-dependent metrics may be needed.

• Adjusting the Search Strategy: EvalML‘s automl.search() API supports several arguments to fine-tune the pipeline search including max_iterations to set the number of rounds, n_jobs to adjust parallelism, and allowed_pipelines to constrain the model/component types.

• Preventing Overfitting: When working with small datasets, it‘s critical to properly validate pipelines to avoid overfitting. EvalML defaults to 3-fold CV but this can be customized via the n_folds and validation_splitter args. Be sure to also use a separate holdout for final model evaluation.

• Interpretability Analysis: To explain and debug a trained pipeline, EvalML integrates with leading interpretability frameworks like SHAP. Use automl.best_pipeline.check_support(‘shap‘) to verify SHAP compatibility and then call automl.best_pipeline.compute_shap_values(X, y) to generate importance scores for prediction analysis.

• Deploying to Production: Once you‘ve found a winning pipeline, you can easily save it for later use in production environments. Calling automl.best_pipeline.save(‘file-path‘) will serialize the entire pipeline to disk. You can then load it back for inference via automl.load_pipeline_from_file(‘file-path‘).

Of course, these are just a few examples of the many levers available to optimize AutoML projects. To truly master a tool like EvalML requires a solid foundation in data science and machine learning concepts. Some key skills to focus on:

• Feature engineering and selection techniques
• Hyperparameter optimization approaches
• Model evaluation, validation, and testing methodologies
• Data preprocessing and sampling methods
• Results analysis and interpretation

The Road Ahead for AutoML

As the field of automated machine learning continues to evolve at a rapid pace, we can expect libraries like EvalML to keep pushing the boundaries of what‘s possible. Some exciting areas of ongoing research and development in AutoML include:

• Neural architecture search (NAS) for optimizing deep learning models [^11]
• Adaptive pipeline search that dynamically adjusts based on dataset characteristics [^12]
• Progressive data ingestion to speed up training on large datasets [^13]
• Causal discovery and inference for generating robust and actionable insights [^14]
• Self-supervised and unsupervised AutoML for unlabeled and unstructured data [^15]

We are truly at an inflection point for democratizing artificial intelligence capabilities. By enabling more individuals across an organization to rapidly build and deploy models with tools like EvalML, data science teams can keep pace with the staggering growth in business data and achieve step-change improvements in productivity.

However, it‘s important to underscore that AutoML is not a replacement for human expertise, but rather a means to augment and scale it. Domain knowledge, results interpretation, and problem formulation remain integral to the success of any data science project. The role of the data scientist is not going away – it‘s evolving to become more focused on high-level experimentation and translating insights into business value.

The road ahead for AutoML is tremendously exciting, with the potential to make AI accessible to every organization in the coming years. As a data scientist looking to stay on the cutting edge, getting hands-on with a library like EvalML is a great way to upgrade your skills while delivering immediate impact. The age of AutoML is here – now is the time to embrace it!

[^1]: The Quant Crunch: How The Demand For Data Science Skills Is Disrupting The Job Market
[^2]: Automated Machine Learning: State-of-the-Art and Open Challenges
[^3]: Enterprise AI/ML Adoption Trends
[^4]: Automated Machine Learning Market Size
[^5]: Benchmarking AutoML Platforms
[^6]: Google‘s AutoML: Cutting Through the Hype
[^7]: Taking the Human Out of the Loop: A Review of Bayesian Optimization
[^8]: A Survey on Evolutionary Machine Learning
[^9]: Early Stopping – But When?
[^10]: 10 Best AutoML Frameworks
[^11]: Neural Architecture Search: A Survey
[^12]: Adaptive Machine Learning Framework
[^13]: Efficient Progressive Data Ingestion for ML
[^14]: CausalAutoML: Causal Inference in AutoML
[^15]: AutoSSL: Towards Automated Semi-Supervised Learning