Salesforce Open Sources TransmogrifAI: Empowering Enterprises with Automated Machine Learning at Scale

The Rise of Auto ML

In recent years, automated machine learning (auto ML) has emerged as one of the hottest trends in the world of data science and artificial intelligence. By intelligently automating many of the tedious and time-consuming tasks involved in building machine learning models, auto ML promises to dramatically accelerate the development process and make machine learning more accessible to a wider range of users and organizations.

The market for auto ML is expected to grow rapidly in the coming years. According to a report from MarketsandMarkets, the global auto ML market size is projected to reach USD 14.8 billion by 2025, up from just USD 269 million in 2019, at a CAGR of 43.7% during the forecast period. This growth is being driven by factors such as the increasing demand for efficient data processing, the need for faster time-to-market, and the shortage of skilled data scientists.

Some of the leading tech companies have invested heavily in auto ML, building powerful platforms and tools to streamline their own internal machine learning workflows. Google has Cloud AutoML, Microsoft has automated ML capabilities in Azure Machine Learning, and Amazon offers SageMaker Autopilot as part of AWS. But until recently, these auto ML platforms were only available as proprietary cloud services.

Over the past year, that has started to change, with a growing number of these tools being released as open source software. The latest and perhaps most significant example of this trend came in August 2018, when Salesforce took the bold step of open sourcing TransmogrifAI, the automated machine learning library that powers its Einstein AI platform.

Introducing TransmogrifAI

TransmogrifAI (pronounced trans-MOG-rih-fye) is an open source auto ML library for structured data, written in Scala and running on top of Apache Spark. It was developed by Salesforce and is used extensively within the company to rapidly build and deploy machine learning models for its various cloud offerings and internal applications.

Key features and capabilities of TransmogrifAI include:

• Automation of feature engineering, model selection, hyperparameter tuning, and model evaluation. The library uses techniques such as heuristic search, meta-learning, and multi-armed bandits to guide the optimization process.
• Scalability to large datasets. Built on Spark, TransmogrifAI can handle terabyte-scale data by distributing computations across a cluster.
• Support for classification and regression on structured data. TransmogrifAI works with data in tabular formats such as CSV files and Parquet.
• Ability to handle messy, incomplete, and high cardinality data. Includes modules for automating data cleaning, imputation, and feature encoding.
• Production-readiness. Models can be exported in formats like PMML and ONNX for deployment.
• Ease of use and flexibility. Offers both a high-level Scala API for end-to-end auto ML and lower-level APIs for fine-grained control.

Under the hood, TransmogrifAI is built around a modular architecture that allows different components to be swapped out or customized. At the core is an automated feature engineering and model training pipeline with three main stages: data processing, feature engineering, and model selection.

The architecture is summarized in this diagram from the TransmogrifAI documentation:

In the data processing stage, raw data is ingested, cleaned, and transformed into a standardized internal format. Automated checks are performed for data quality issues such as missing values, outliers, and inconsistent formats. Relevant metadata is also extracted to inform downstream processing.

Next, in the feature engineering stage, a library of scalable feature transformers is applied to automatically generate a rich set of candidate features. This includes transformations such as arithmetic operations, aggregations, category encodings, text processing, and more. Techniques like lasso regression are used for feature selection to identify the most informative set of features.

Finally, in the model selection stage, a collection of candidate models are trained and evaluated on the engineered feature set. This can include various linear and tree-based algorithms such as logistic regression, decision trees, random forests, and gradient boosted trees. Bayesian optimization and multi-armed bandits guide the hyperparameter tuning to efficiently search for the best performing models.

At the end of this pipeline, the most performant model is selected and can be exported for production deployment and monitoring. The entire process is automatic and can be run end-to-end with just a few lines of code, while still allowing the user to customize the pipeline at any stage if desired.

To get a sense of what this looks like in practice, here is an example from the TransmogrifAI quick start guide. This snippet shows how to train a binary classification model:

import com.salesforce.op._
import com.salesforce.op.readers._
import com.salesforce.op.features._
import com.salesforce.op.stages.impl.classification._

// Read data from a CSV file
val schema = StructType(
  StructField("id", StringType, nullable = true) ::
  StructField("text", StringType, nullable = true) ::
  StructField("clicked", DoubleType, nullable = true) :: Nil
)
val dataset = DataReaders.Simple.csvCase[Record](path = pathToData, schema).read()

// Extract label and text features
val label = FeatureBuilder.RealNN[Record].extract(_.clicked.toRealNN).asResponse
val textFeatures = FeatureBuilder.Text[Record].extract(_.text.toText).asPredictor

// Automated feature engineering
val featureVector = textFeatures.vectorize(numHashes = 10000)

// Automated feature selection
val checkedFeatures = label.sanityCheck(featureVector, checkSample = 1.0, removeBadFeatures = true)  
val cleanFeatures = checkedFeatures.logTransform()

// Automated model selection and training
val prediction = BinaryClassificationModelSelector().setInput(label, cleanFeatures).getOutput()
val model = prediction.train()

// Model evaluation and export
val evaluator = Evaluators.BinaryClassification()
val metrics = evaluator.setLabelCol(label).setPredictionCol(prediction).evaluate(model)
val bestModel = model.bestModel.asInstanceOf[BinaryClassificationModel]
bestModel.save(pathToModel)

The TransmogrifAI team at Salesforce has reported numerous benefits from using this library internally across a variety of customer-facing and internal use cases. Some key results include:

• Significant reduction in model development time. For one client project, the use of TransmogrifAI reduced the modeling process from months to just 2 weeks, an 86% reduction in time to value.
• Performance lift over human-tuned models. For a sales forecasting use case, TransmogrifAI found a model that achieved a 15% improvement in accuracy over the best manually-tuned model.
• Increased productivity for data science teams. By automating the repetitive parts of the workflow, data scientists are able to spend more of their time on high-value tasks like understanding business context and interpreting results.

Open Sourcing Innovation

In open sourcing TransmogrifAI, Salesforce has made a powerful tool available to the broader community and expanded access to cutting-edge auto ML capabilities. Data scientists and developers can now leverage the work that has gone into the library and adapt it to their own projects and use cases.

The TransmogrifAI repository on GitHub includes extensive documentation and examples to help users get up and running quickly. Key resources include:

• Quick start and installation guide
• User guide covering concepts, APIs, and workflow
• Scaladoc API reference
• Example notebooks and datasets

There is also an active community of developers and users that has formed around the project. Salesforce hosts a public Slack workspace where users can connect, a Google group for announcements and discussions, and a Twitter account (@TransmogrifAI) sharing project updates.

Salesforce has been investing in sustaining and growing this community through a variety of initiatives. Members of the TransmogrifAI engineering team frequently give talks and host workshops at conferences and meetups. The team also launched a blog featuring technical deep dives, case studies, and perspectives on auto ML best practices.

This commitment to openness and community building around TransmogrifAI is part of Salesforce‘s broader embrace of open source software. Salesforce is a founding member of the Constellation open source foundation for big data and AI, and has released several other major projects like Argus for time series monitoring and TransmogrifOCR for optical character recognition.

By sharing its tools and knowledge, Salesforce aims to spur innovation in machine learning and AI across the ecosystem. "No one company can solve all the challenges that will emerge as AI becomes more embedded in our daily lives," says Shubha Nabar, Senior Director of Data Science at Salesforce. "By open sourcing TransmogrifAI and engaging with the community, we want to play a part in moving the technology forward in a responsible and sustainable way."

The Future of Auto ML

The growing availability of open source tools like TransmogrifAI, along with other auto ML libraries like auto-sklearn, TPOT, and Auto-Keras, has the potential to be a real game changer for applied machine learning. By making powerful automation capabilities more widely accessible, these projects can democratize and accelerate the development of intelligent applications.

Some of the key opportunities and benefits of the proliferation of auto ML include:

• Enabling non-experts and citizen data scientists. Lowering the barriers to building models and deriving insights from data.
• Freeing up experts to work on higher-level problems. Letting data scientists and machine learning engineers focus on novel applications vs. repetitive tasks.
• Accelerating model velocity and deployment. Shortening time from prototype to production and increasing agility.
• Enhancing efficiency and scalability. Optimizing resource utilization and supporting very large datasets and workflows.
• Promoting standardization and best practices. Codifying and sharing proven techniques for tasks like model evaluation and monitoring.

At the same time, the rise of auto ML also surfaces important challenges and risks that will need to be addressed:

• Ensuring proper usage and preventing misuse. Providing guard rails and education for inexperienced users of auto ML tools.
• Maintaining human agency and accountability. Preserving transparency and control over automated modeling processes and decisions.
• Detecting and mitigating unintended consequences. Proactively identifying potential issues like bias, privacy violations, and concept drift.
• Integrating auto ML with responsible AI practices. Applying ethical principles and governance frameworks to auto ML workflows and outputs.
• Validating auto ML systems and explaining their results. Developing robust techniques for testing and interpreting models produced by auto ML.

Commercial and open source auto ML solutions like TransmogrifAI will undoubtedly continue to advance rapidly, expanding to new data modalities, use cases, and ML techniques. Realizing their full potential in the real world will require ongoing collaboration between diverse stakeholders to put the right organizational and societal constructs in place.

In the nearer term, I expect auto ML to be increasingly adopted in the enterprise as a core component of the data science and machine learning stack. A 2020 survey by Algorithmia found that 50% of organizations are already using or plan to use auto ML tools. And a Forrester report argues that "automated machine learning is a must-have in data science platforms."

Companies that embrace auto ML to augment their human expertise will be able to innovate faster, harnessing ever-larger volumes of data for business advantage. Open source libraries like TransmogrifAI level the playing field, putting cutting-edge capabilities in the hands of a wider range of organizations.

Looking further ahead, it‘s intriguing to consider a world in which auto ML becomes the norm rather than the exception. What role will data scientists play in an era of pervasive automation? Some speculate that auto ML will eliminate the majority of traditional data science jobs. But many experts believe that demand for data science skills will continue to grow, even as the nature of the work evolves.

Rather than replacing human intelligence, auto ML will shift data scientists to higher-value tasks that are complementary to AI, like formulating problems, interpreting results, and translating insights into action. As Francesca Lazzeri of Microsoft puts it in a recent article, "data scientists will increasingly become teachers and subject-matter experts rather than tool jockeys."

Others predict that data science roles will specialize and proliferate. An article in InfoWorld envisions titles like "machine learning model operations engineer, automated machine learning architect, machine learning reliability engineer, and ethical machine learning expert." Auto ML serves as a foundation for this diversification of skills.

Of course, the full implications of auto ML on the future of work remain to be seen. What I do feel confident about is that automation will be an essential tool for data scientists and organizations of all sizes and levels of AI maturity in the years ahead. And open source auto ML libraries like Salesforce‘s TransmogrifAI are helping to blaze the trail, advancing the state of the art and expanding access to powerful machine learning capabilities.

The key will be to continue to evolve these frameworks in responsible, human-centric ways, balancing the transformative potential of auto ML with strong governance and ethical practices. By doing so, we can build a future in which AI truly augments and empowers people to achieve more than ever before.