Intel Open Sources nGraph: Enabling Data Scientists to Focus on Model Development, Not Hardware and Frameworks

Introduction

The rapid evolution of artificial intelligence (AI) and machine learning (ML) has transformed nearly every industry, from healthcare and finance to transportation and entertainment. However, the proliferation of frameworks and hardware platforms has also introduced new challenges for data scientists and developers looking to build and deploy cutting-edge deep learning models.

To help address these challenges, Intel has open sourced nGraph, a powerful toolkit that abstracts away the complexities of specific hardware and frameworks, allowing data scientists to focus on model development and performance optimization. By providing a framework-neutral intermediate representation (IR) and a suite of compiler tools, nGraph enables faster iteration, easier portability, and better performance across a wide range of platforms.

How nGraph Works: A Deep Dive

At its core, nGraph is a C++ library and runtime / compiler suite that converts deep neural network models from popular frameworks into a hardware-agnostic intermediate representation. This IR is then optimized for the target hardware platform, with the resulting code compiled and executed using platform-specific libraries and drivers.

The nGraph IR is designed to be highly flexible and expressive, with support for a wide range of operators and data types. This allows it to represent complex models from different domains, including image classification, natural language processing, and recommendation systems.

To generate optimized code for a particular hardware target, nGraph uses a multi-stage compilation process. First, the model is ingested from the source framework and converted into the nGraph IR. This IR is then passed through a series of graph-level optimizations, which perform tasks like operator fusion, memory layout optimization, and data type quantization.

Next, the optimized IR is lowered to a platform-specific representation, such as LLVM IR for CPUs or CUDA for NVIDIA GPUs. This representation is then compiled using the appropriate backend compiler, with additional platform-specific optimizations applied. Finally, the resulting binary code is loaded and executed using the nGraph runtime, which manages memory allocation, data transfer, and kernel execution.

One of the key advantages of nGraph‘s architecture is that it allows for easy experimentation with different optimization techniques and strategies. For example, data scientists can use nGraph‘s profiling tools to identify performance bottlenecks in their models, then apply different graph- and kernel-level optimizations to improve throughput or reduce latency.

nGraph also provides a Plugin API that allows hardware vendors to add support for new devices and custom accelerators. This makes it possible to target emerging platforms like neuromorphic chips and FPGAs, as well as specialized ASICs for edge and mobile deployment.

Supported Frameworks and Integration

As of 2023, nGraph supports several popular deep learning frameworks, including:

• TensorFlow: nGraph provides a drop-in replacement for the TensorFlow backend, allowing users to run existing TensorFlow models on Intel hardware with minimal code changes. nGraph also integrates with the TensorFlow Serving platform for optimized model inference.

• PyTorch: nGraph supports PyTorch models through the PyTorch C++ frontend, which can be used to convert PyTorch models to the nGraph IR. This allows PyTorch users to leverage nGraph‘s performance optimizations and hardware support.

• MXNet: nGraph integrates with the MXNet framework through the MXNet Bridge, which maps MXNet operators to nGraph IR ops. This allows MXNet users to target Intel CPUs and other supported hardware backends.

• ONNX: nGraph can ingest models in the Open Neural Network Exchange (ONNX) format, which provides a common representation for models from different frameworks. This makes it easier to deploy models trained in various frameworks to Intel hardware.

nGraph also has experimental support for frameworks like CNTK and PaddlePaddle, with plans to expand coverage to additional frameworks in the future.

To use nGraph with a supported framework, data scientists typically need to make only minimal code changes, such as specifying nGraph as the backend using a framework-specific API or environment variable. nGraph then takes care of the rest, automatically converting the model to the nGraph IR and applying the necessary optimizations for the target hardware.

Performance Comparison

One of the key benefits of nGraph is its ability to deliver optimized performance across a range of hardware platforms without the need for extensive hand-tuning. In many cases, simply switching to nGraph as the backend for an existing model can provide significant speedups.

For example, consider the following performance comparison of nGraph vs. the default TensorFlow backend on a set of popular image classification models:

As we can see, nGraph delivers speedups of up to 1.39x on these models, without any changes to the model code itself. This is made possible by nGraph‘s advanced graph optimizations and kernel-level tuning, which can extract maximum performance from the underlying hardware.

Similarly, nGraph has been shown to deliver significant performance gains on natural language processing workloads like text classification and machine translation. For example, in one study of the Transformer model on the WMT14 English-German dataset, nGraph achieved a 1.8x speedup over the native TensorFlow backend on an Intel Xeon Platinum 8180 CPU [1].

Of course, the actual performance gains from using nGraph will depend on the specific model, batch size, and hardware configuration. However, these examples demonstrate the potential for nGraph to accelerate a wide range of deep learning workloads without the need for extensive model-specific optimization.

Intel‘s AI Strategy and Ecosystem

nGraph is just one piece of Intel‘s broader strategy to accelerate the adoption of AI and machine learning across industries. As part of this effort, Intel has developed a comprehensive portfolio of hardware and software solutions designed to support the end-to-end AI pipeline, from data preparation and model training to inference and deployment.

On the hardware side, Intel offers a range of processors and accelerators optimized for AI workloads, including:

• Intel Xeon Scalable Processors: Intel‘s flagship CPU platform, which features advanced vector extensions (AVX-512) and support for large memory configurations.
• Intel Nervana Neural Network Processors (NNPs): Purpose-built ASICs for training and inference, designed to deliver high performance and efficiency for deep learning workloads.
• Intel FPGAs: Programmable accelerators that can be customized for specific AI applications and models.
• Intel Movidius VPUs: Low-power vision processing units designed for edge AI deployments in cameras, drones, and other devices.

To help data scientists take full advantage of this hardware, Intel has also developed a suite of optimized software libraries and tools, including:

• Intel oneAPI AI Analytics Toolkit: A collection of Python libraries and frameworks for data analytics and machine learning, optimized for Intel architecture.
• Intel Distribution of OpenVINO Toolkit: A set of tools and libraries for optimizing and deploying computer vision and deep learning inference applications at the edge.
• Intel Optimization for TensorFlow and PyTorch: Optimized distributions of popular deep learning frameworks that are tuned for performance on Intel CPUs and GPUs.

nGraph plays a key role in this ecosystem by providing a unified compiler and runtime that can target multiple hardware backends and integrate with multiple frameworks. This allows data scientists to develop and optimize models using their preferred tools and libraries, while still taking advantage of Intel‘s advanced hardware capabilities.

Competitive Landscape and Positioning

Intel is not alone in its efforts to simplify and accelerate AI development across diverse hardware and software platforms. In recent years, several other major technology companies and open-source projects have introduced tools and frameworks that aim to address similar challenges.

One of the most prominent of these is ONNX (Open Neural Network Exchange), an open format for representing deep learning models that enables interoperability between different frameworks and tools. Initially developed by Facebook and Microsoft, ONNX has since been adopted by a wide range of companies and projects, including PyTorch, AWS, and Azure Machine Learning.

While nGraph supports ingesting models in the ONNX format, it differs from ONNX in several key ways. First and foremost, nGraph is a full-stack compiler and runtime for deep learning, with support for multiple hardware backends and advanced graph and kernel optimizations. ONNX, in contrast, focuses primarily on providing a common model representation format, with less emphasis on performance optimization.

Another major player in this space is NVIDIA, which has developed its own suite of AI software tools and libraries, including TensorRT for optimized inference on NVIDIA GPUs, and the CUDA Deep Neural Network library (cuDNN) for accelerated deep learning primitives. While these tools are tightly integrated with NVIDIA‘s own hardware, nGraph offers a more flexible and portable solution that can target multiple hardware backends.

Other notable projects in this space include Apache TVM, an open-source compiler stack for deep learning that targets a wide range of hardware backends, and Google‘s XLA (Accelerated Linear Algebra), a domain-specific compiler for linear algebra operators that can accelerate TensorFlow models on CPUs and GPUs.

Compared to these tools, nGraph stands out for its broad framework and hardware support, as well as its powerful IR and flexible plugin architecture. As an open-source project backed by Intel, nGraph also benefits from significant engineering resources and a robust roadmap for future development.

Future Directions and Potential Applications

As the AI landscape continues to evolve at a rapid pace, nGraph will play an increasingly important role in enabling data scientists to take advantage of the latest hardware and software technologies. Some key areas of future development and potential applications include:

• Expanded support for edge and mobile deployment: As more AI workloads move to the edge, nGraph could help data scientists optimize and deploy models on resource-constrained devices like smartphones, drones, and IoT sensors.

• Automated model optimization and architecture search: By integrating with tools for neural architecture search and hyperparameter optimization, nGraph could help automate the process of designing and tuning models for specific hardware targets and performance requirements.

• Integration with AI DevOps and MLOps platforms: As organizations adopt more sophisticated workflows for developing and deploying AI applications, nGraph could be integrated with popular DevOps and MLOps platforms like Kubeflow and MLflow to enable seamless model optimization and deployment.

• Compiler optimizations for sparse and quantized models: To help reduce the memory footprint and computational cost of deep learning models, nGraph could be extended with more advanced support for model compression techniques like pruning, quantization, and sparsification.

• Neuromorphic computing and other emerging architectures: As new hardware architectures like neuromorphic chips and analog accelerators mature, nGraph could provide a way to abstract away their unique programming models and enable data scientists to leverage their capabilities.

Ultimately, the goal of nGraph is to democratize AI development and make it easier for data scientists and researchers to focus on the high-level tasks of model design and performance optimization, without getting bogged down in the details of specific hardware and software platforms. By providing a flexible, portable, and performant compiler and runtime for deep learning, nGraph is well-positioned to help drive the next wave of AI innovation and adoption.

Conclusion

The open-sourcing of nGraph by Intel represents a significant milestone in the evolution of AI and machine learning tooling. By providing a framework-neutral IR and a powerful compiler and runtime stack, nGraph enables data scientists to focus on model development and performance optimization, rather than the intricacies of specific hardware and software platforms.

With support for popular frameworks like TensorFlow and PyTorch, as well as a wide range of Intel and third-party hardware backends, nGraph offers a flexible and portable solution for accelerating deep learning workloads across the AI pipeline. Its modular architecture and extensible plugin system also make it well-suited to adapt to the rapidly changing landscape of AI hardware and software.

As more organizations look to adopt AI and machine learning to drive innovation and efficiency, tools like nGraph will play an increasingly important role in enabling data scientists and developers to build and deploy cutting-edge models at scale. By abstracting away the complexities of hardware and software integration, nGraph helps lower the barriers to entry for AI development and accelerate the pace of innovation across industries.

Looking ahead, the future of AI tooling will likely be shaped by several key trends, including the continued proliferation of specialized hardware accelerators, the growth of edge and mobile deployment, and the adoption of more sophisticated DevOps and MLOps practices. By providing a flexible and performant foundation for deep learning development, nGraph is well-positioned to help organizations navigate these trends and realize the full potential of AI and machine learning.

Ultimately, the success of nGraph and other similar tools will depend on their ability to build and sustain a vibrant open-source community of developers, researchers, and end-users. By collaborating with partners across the AI ecosystem and investing in ongoing innovation and optimization, Intel has the opportunity to establish nGraph as a key enabler of the next generation of AI applications and services.