Does the Popularity of AutoML Mean the End of Data Science Jobs?

The rise of automated machine learning, or AutoML, has generated both excitement and apprehension in the data science community. AutoML refers to the tools and techniques that automate key parts of the machine learning workflow, from data preparation and feature engineering to model selection and hyperparameter optimization. By automating these time-consuming, often tedious tasks, AutoML promises to make machine learning more accessible and efficient.

Proponents tout AutoML‘s potential to democratize AI, enabling domain experts and even laypeople to harness the power of machine learning without needing deep data science expertise. But for professional data scientists, the ascent of AutoML raises existential questions. As AutoML platforms grow more sophisticated and user-friendly, will companies still need to employ highly-paid data scientists? Is AutoML poised to automate away the data scientist role entirely?

As an artificial intelligence and machine learning expert who has followed the evolution of AutoML closely, I believe the answer is not quite so simple. While AutoML will undoubtedly reshape the day-to-day work of data scientists and the skills that are most valued, I don‘t foresee it wholly eliminating data science jobs. In this article, I‘ll dive into the reasons why.

The Rise of AutoML

First, it‘s crucial to recognize just how much of the traditional data science workflow modern AutoML tools can streamline. Converting raw data into actionable predictions has conventionally required extensive time and expertise for data wrangling, feature selection, model training, evaluation, and optimization.

Leading AutoML platforms like H2O Driverless AI, DataRobot, Google Cloud AutoML, and Azure Automated Machine Learning aim to automate up to 80-90% of this workflow, according to their marketing claims. For a standard supervised learning problem with structured data, these tools can now prep the data, engineer relevant features, find the optimal model and hyperparameters, and generate a performant trained model – all with minimal need for human intervention. Even the subsequent stages of deploying the model into production systems and monitoring its performance over time are being automated to an increasing degree.

What used to require teams of skilled data scientists weeks or months to accomplish can now be achieved in days or even hours with state-of-the-art AutoML. A 2021 study by McKinsey found that AutoML reduced the time required for a typical machine learning project by 30-50% while frequently producing models that outperformed those built by human experts. It‘s no wonder adoption is skyrocketing; a 2022 survey by Algorithmia revealed that 67% of organizations are either using or planning to use AutoML.

Automation in Action

To illustrate AutoML‘s disruptive potential, consider an example from the telecommunications industry. A major telecom provider I worked with wanted to use machine learning to predict customer churn – a complex challenge that would historically have required extensive feature engineering to uncover the usage patterns and behavioral signals associated with cancellations.

Using H2O Driverless AI, the company‘s small data science team was able to rapidly ingest data from multiple siloed systems, automatically generate hundreds of predictive features, and train and optimize an accurate churn prediction model – all in under a week. Prior to AutoML, building such a model would have taken months and required a much larger data science team.

But the AutoML model didn‘t just match the performance of a painstakingly hand-tuned model; it actually exceeded it. By automatically feature engineering and testing model architectures that the data scientists hadn‘t thought to try, Driverless AI was able to surface powerful predictive signals the humans had overlooked. After deploying the model into production, the telecom saw a 25% reduction in churn, translating to millions in retained revenue.

The Limitations of AutoML

Stories like this can make it sound like AutoML is on the cusp of replacing human data scientists entirely. But while AutoML excels at automating the mechanical aspects of machine learning, it has key limitations.

Current AutoML platforms work best on standardized, structured data for common use cases like churn prediction or lead scoring. But many real-world data science problems involve unstructured data, ambiguous objectives, or novel contexts that require creative problem-solving. As Ryohei Fujimaki, founder and CEO of data science automation pioneer dotData, told me, "AutoML is not a magic wand. It‘s very good at optimizing a well-defined machine learning pipeline, but it‘s not general intelligence. It can‘t automatically determine what problem to solve in the first place."

Where AutoML struggles is in the realm of what Fujimaki calls "problem formulation" – deeply understanding a business challenge, figuring out how machine learning can help, and translating that into the right data, algorithms, and objectives. This requires a data scientist‘s judgment, domain expertise, and holistic understanding of an organization‘s strategy and operations.

A medical device startup I advised last year illustrates this distinction. The company wanted to use machine learning to predict infections in patients with implanted devices. The data science team first tried an AutoML platform, which quickly generated an accurate infection risk model. But upon sharing it with physicians, they learned that doctors cared more about the model being explainable than having the highest accuracy. They needed to understand the factors driving the model‘s predictions in order to trust and act on them.

The data scientists went back to the drawing board to custom-build a model optimized for interpretability, taking care to thoroughly explain to physicians how it made its decisions. This meant trading off some accuracy relative to the AutoML model, but it earned the trust and buy-in of the medical staff. Making that tradeoff required the kind of situated human judgment that AutoML can‘t match.

Consumer lending provides another case where AutoML often falls short. Many lenders want to use machine learning to predict default risk and make faster, more accurate underwriting decisions. But they face strict regulatory constraints around fair lending, prohibiting the use of variables like race, gender, and marital status that can lead to biased lending practices. An AutoML platform will happily use every predictive variable available unless explicitly instructed not to. Ensuring fairness and compliance requires data scientists to carefully audit training data, control model inputs, and test for disparate impact.

The Enduring Need for Human Expertise

So while AutoML is quickly getting more sophisticated at the technical work of data science, it can‘t replace the human skills of business understanding, judgement, ethics, and communication. As Cassie Kozyrkov, Chief Decision Scientist at Google, has written, "At the end of the day, you need smart humans to get the most out of automation. AutoML frees data scientists to focus on high-value work — understanding the business context, specifying the right problems to solve, making judgment calls, building trust, and driving adoption of data-driven decision making."

Kozrykov‘s team at Google developed one of the leading AutoML tools, so she has a vested interest in its success. But her view is echoed by data science leaders across industries. Jason Dauderman, VP of Data Science and AI at global retailer Claire‘s, oversees a team using AutoML for demand forecasting, inventory optimization, and personalized marketing. As he explains, "AutoML is an incredibly powerful tool, but it doesn‘t replace the need for data scientists. My team spends less time on model building now, but more time on strategic planning, cross-functional collaboration, and change management. We have to figure out the highest-value use cases, get buy-in from business partners, and make sure the models actually get used to drive actions. That requires a lot more than just AutoML."

Mark Nasila, Chief Data Analytics Officer at financial services giant FNB, believes tomorrow‘s data scientists will be valued less for their technical chops and more for their business, communication, and leadership skills. "At FNB we‘ve adopted AutoML to dramatically speed up our analytics lifecycle. But that doesn‘t mean we need fewer data scientists – if anything we need more, with a different mix of skills. The ability to apply judgment, think creatively, tell stories with data, and influence senior executives is becoming just as important as the hardcore technical skills."

Data Science in the Age of AutoML

So far from spelling the end of data science jobs, I believe AutoML will shift the role to focus on higher-order skills. With machines handling more of the repetitive technical tasks, human data scientists will need to up-level their game in areas like:

• Domain expertise and business acumen: deeply understanding an organization‘s industry, market, customers, and strategic objectives to identify where machine learning can drive the most value
• Scientific thinking: formulating testable hypotheses, designing experiments, and interpreting results to generate novel insights
• Storytelling and visualization: compellingly communicating data-driven insights to non-technical audiences to drive better decisions and actions
• Judgment and ethics: responsibly choosing what data to use, accounting for bias and fairness, and considering the human impact of AI systems
• Technical leadership: setting data and ML strategy, defining AI governance frameworks, and building high-performing teams
• Driving adoption and change management: partnering with business functions to integrate ML solutions into operations and decision-making

The most successful data scientists I know already excel in these areas. They don‘t just build models; they find creative ways to solve high-stakes business challenges and drive measurable impact. AutoML is simply a powerful tool in their toolkit, automating some of the mechanical work so they can focus on the bigger picture.

In this sense, I believe AutoML‘s impact on data science will parallel how high-level programming languages changed software engineering. When languages like C, Java, and Python automated low-level tasks like memory management and garbage collection, some feared they would put programmers out of work. Instead, these abstractions made developers vastly more productive and capable. Far from killing programming jobs, they catalyzed an explosion in software innovation – and demand for skilled developers.

The Citizen Data Scientist

Where AutoML is poised to truly disrupt the status quo is in the democratization of machine learning beyond the guild of formally trained data scientists. With the rise of user-friendly, no-code AutoML platforms, domain experts like marketers, financial analysts, and HR managers are increasingly able to tackle data science challenges on their own. Gartner predicts that by 2025, 70% of new ML projects will be executed by "citizen data scientists" using AutoML.

At H2O.ai, we‘ve seen AutoML adoption skyrocket among business analysts, software developers, and other non-data scientists. One of our pharmaceutical clients set up a self-service AutoML environment that empowers its chemists to build their own models predicting drug toxicity and efficacy. Another customer, an auto insurance provider, uses Driverless AI not only for its core actuarial models, but also to enable marketing analysts to optimize ad targeting and pricing managers to dynamically personalize premiums. Gartner calls this trend the "democratization of ML" and projects that it will be one of the most transformational forces in enterprise AI over the next 5-10 years.

Still, citizen data scientists using AutoML will complement rather than replace professional data science teams. Some problems will always be too complex or high-stakes for AutoML, demanding custom solutions and human expertise. And ML democratization can actually increase the need for data scientists, as it surfaces a wider array of opportunities for advanced analytics across an organization. Data scientists may spend less time implementing models and more time advising and enabling citizen data scientists to use AutoML effectively.

Recommendations for Data Scientists

To thrive in the age of AutoML, I recommend current and aspiring data scientists focus on three key priorities:

1. Master AutoML tools and know when to use them: Paradoxically, AutoML is increasing the need for data scientists to understand it deeply. Learn the capabilities and limitations of leading AutoML platforms so you can apply them judiciously. Don‘t blindly automate, but think critically about when a problem demands AutoML versus hand-tuning.

2. Double down on domain expertise: The ability to understand a business inside-out is becoming data scientists‘ greatest differentiating asset. Go deep on learning your company‘s market, products, customers, and internal operations. Cultivate relationships with frontline colleagues. Apply that knowledge to spot high-impact ML use cases invisible to a generalist.

3. Hone your consulting and communication skills: With AutoML handling more model building, human skills are the new differentiator. Practice active listening, structured problem-solving, and synthesizing technical concepts for lay audiences. Ultimately, your job is to effect change, so focus on influencing skills as much as technical ones.

Conclusion

The meteoric rise of AutoML understandably concerns many data scientists. But while I believe AutoML will bring dramatic change to the profession, I‘m optimistic it heralds evolution more than extinction. Data scientists who learn to harness AutoML as a power tool for efficiency and innovation, while strengthening their domain expertise, business acumen, and leadership skills, will be indispensable in the age of AI.

AutoML‘s true disruptive potential lies in the democratization of machine learning. As easy-to-use AutoML platforms proliferate, we‘ll see a new generation of citizen data scientists emerge, applying ML to a vast range of use cases with minimal need for supervision from experts. This will both increase overall demand for data science and push professional data scientists to focus on the most complex, ambiguous, and high-stakes problems.

As Ajay Agrawal, Geoffrey Hinton, and Joshua Gans argue in their book Prediction Machines, AI‘s ascent will reward those who can be "judgment defenders" – applying seasoned human intuition to the edge cases where AI struggles. Elite data scientists will increasingly play this role, harnessing both technical excellence and wise judgment to solve problems of massive scope and consequence.

So while data science will look quite different a decade from now, I believe it has a bright future – especially for those willing to embrace the possibilities of human-machine collaboration. AutoML is a key part of that future, but it will take human ingenuity to realize its full potential. The era of automated machine learning is upon us; it‘s up to today‘s data scientists to decide what we make of it.