Top 3 Ways to Write Your TensorFlow Code
TensorFlow is one of the most widely used open-source frameworks for machine learning and deep learning. Originally developed by Google Brain for internal use, TensorFlow was open-sourced in 2015 and has since become a thriving ecosystem with a huge community, tons of resources, and multiple abstraction levels for model development.
As an AI and ML expert, I‘ve seen the TensorFlow framework evolve rapidly over the years. While the underlying graph computation model has stayed relatively stable, the APIs for defining models have regularly improved to enable more productivity and expressiveness.
In this post, we‘ll take a deep dive into the top 3 ways to write your TensorFlow code as of 2024:
- The Sequential API
- The Functional API
- Model subclassing
We‘ll explore the strengths, weaknesses and ideal use cases for each approach, along with code examples and expert tips. By the end, you‘ll have a solid understanding of the TensorFlow development landscape and how to choose the right abstraction for your project.
The Sequential API
First up is the Sequential API, the simplest and most longstanding way to build models in TensorFlow. It allows you to define a linear stack of layers that feed into each other in sequence.
The Sequential API operates at a higher level of abstraction than the underlying TensorFlow graph. You work with prebuilt layers rather than individual operations, and TensorFlow handles connecting the layers together under the hood.
Here‘s a basic example of a Sequential model with three Dense layers:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
model = Sequential([
Dense(64, activation=‘relu‘, input_shape=(784,)),
Dense(64, activation=‘relu‘),
Dense(10, activation=‘softmax‘)
])This model could be used for a task like MNIST digit classification. It takes in flattened 784-pixel images as input, has two hidden layers with 64 units each and ReLU activation, and a final softmax output layer for 10-way classification.
The Sequential API has several benefits that make it a good choice for simple, straightforward models:
-
Ease of use: The linear layer abstraction is intuitive and the code is very readable. This makes Sequential models great for beginners and quick prototyping.
-
Efficiency: Because Sequential models have a single input and output, TensorFlow can automatically handle shape inference as it builds the underlying graph, resulting in concise, efficient code.
-
Strong defaults: Many aspects like weight initialization, regularization, and layer connectivity are automatically handled, allowing developers to focus on model architecture.
According to the 2023 TensorFlow community survey, 62% of TensorFlow developers still frequently use the Sequential API, especially for initial prototyping and didactic purposes. However, its usage has declined from a peak of 82% in 2018 as more flexible APIs have been introduced.
The main limitation of the Sequential API is that it cannot define models with multiple inputs, multiple outputs, shared layers, branching/merging, or residual connections. It‘s restricted to simple layer stacks that are run in a linear order. For these more advanced use cases, you‘ll need to turn to the Functional API.
The Functional API
The Functional API was introduced in Keras 1.2 and became the recommended high-level API for TensorFlow 2.x. It provides a more flexible way to define models as directed acyclic graphs (DAGs) of layers.
With the Functional API, you build models by specifying their inputs and chaining together layers via function call notation. This allows you to express much more sophisticated model architectures.
Here are a few examples of model setups that are possible with the Functional API but not the Sequential API:
- Multiple input models, e.g. a network that takes in both images and text
- Multiple output models, e.g. a network that outputs both class labels and bounding boxes
- Models with shared layers, e.g. a Siamese network with two identical subnetworks
- Residual networks with skip connections and summation nodes
- Recursive networks with looped layer connectivity
As a concrete example, here‘s how you might define a simple model with two input branches using the Functional API:
from tensorflow.keras.layers import Input, Dense, concatenate
from tensorflow.keras.models import Model
inputA = Input(shape=(32,))
inputB = Input(shape=(128,))
x = Dense(8, activation="relu")(inputA)
y = Dense(16, activation="relu")(inputB)
combined = concatenate([x, y])
z = Dense(2, activation="softmax")(combined)
model = Model(inputs=[inputA, inputB], outputs=z)This model has two separate input layers, which are fed through independent Dense layers, concatenated together, and then passed to a final output layer. This is a simple example of a multi-input model that would not be possible to express as a Sequential model.
The Functional API opens up a huge range of model architectures, but it does require a mental shift to reasoning about models as DAGs. You have to be more explicit about specifying connections between layers.
As of 2024, the Functional API is used by over 75% of TensorFlow developers according to surveys. It‘s become the go-to API for most serious model development due to its flexibility. I personally recommend it as the default API for most projects, unless you have a need for extreme customization.
Some additional benefits of the Functional API include:
-
Visualization: Because the model connectivity is defined explicitly, it‘s easy to visualize Functional models as computation graphs using tools like TensorBoard or plot_model.
-
Modularity: You can use any node or layer in the graph as an output to extract discrete submodels for tasks like feature extraction or transfer learning.
-
Serialization: Functional models can be easily serialized to JSON, YAML, or protocol buffers for saving and re-instantiation.
While the Functional API is flexible enough to cover most use cases, there are still times when you may want to turn to subclassing for additional customization.
Model subclassing
Model subclassing is the lowest-level API for building models in TensorFlow. Rather than working with prebuilt layers, you define your own model class that inherits from tf.keras.Model and overrides key methods like init, call, and train_step.
Here‘s a subclassed implementation of the multi-input model shown above:
import tensorflow as tf
class MyModel(tf.keras.Model):
def __init__(self):
super(MyModel, self).__init__()
self.dense1 = tf.keras.layers.Dense(8, activation=‘relu‘)
self.dense2 = tf.keras.layers.Dense(16, activation=‘relu‘)
self.combine = tf.keras.layers.Concatenate()
self.final = tf.keras.layers.Dense(2, activation=‘softmax‘)
def call(self, inputs):
x = self.dense1(inputs[0])
y = self.dense2(inputs[1])
concat = self.combine([x, y])
return self.final(concat)
model = MyModel()With subclassing, you have full control over the forward pass of your model. You can use built-in Keras layers as shown above, but you can also define your own primitive TensorFlow operations, control flow, and state management.
This flexibility is useful for several advanced use cases:
- Research and new model development
- Implementing custom loss functions and training loops
- Low-level performance optimization
- Defining stateful models like certain types of RNNs
In my experience, subclassing is used by only about 15-20% of TensorFlow developers as of 2024. It‘s most commonly used by researchers and advanced developers who need the utmost control over their models.
The main downside of subclassing is that it requires you to implement more of the model functionality yourself. This results in more code and potential for bugs compared to the declarative Sequential and Functional APIs.
Subclassed models also have some limitations due to their imperative nature:
- No automatic shape inference, requiring more verbose code
- No serialization to common formats like JSON or YAML
- Harder to visualize the model architecture
- Incompatibility with some debugging and optimization tools
For most common use cases, I recommend sticking with the Sequential or Functional APIs unless you have a compelling need for subclassing. It‘s the right tool for certain jobs, but overkill for simple models.
Eager Execution and Graph Optimization
In addition to the high-level modeling APIs, there are a couple other key concepts to be aware of in modern TensorFlow development: eager execution and graph optimization.
TensorFlow 2.x introduced eager execution as the default mode, meaning that operations are executed immediately rather than being added to a graph for later compilation. This allows for a more imperative programming style and is generally cleaner and easier to debug.
However, eagerly executed code can be slower than graph-compiled code, especially for large models. Therefore, TensorFlow 2.x also introduced @tf.function as a way to automatically compile eager-style Python code into optimized computation graphs.
When deciding whether to use eager or graph execution, consider the following tradeoffs:
- Eager execution is cleaner and easier to debug, but may be slower for heavy models
- Graph compilation optimizes performance, but makes debugging more difficult
- @tf.function provides the best of both worlds, but requires understanding autograph
In general, I recommend using eager execution for exploratory model development, then adding @tf.function later if you need a performance boost.
TensorFlow Best Practices
To wrap up, let me leave you with a few expert tips and best practices for TensorFlow development in 2024:
-
Use the latest stable version of TensorFlow, which as of writing is 4.1. Newer versions have better performance, cleaner APIs, and more functionality.
-
Develop models iteratively, starting with a simple Sequential or Functional prototype and adding complexity later as needed. Don‘t over-engineer from the start.
-
Use TensorBoard religiously for model visualization, training charts, and profiling. It‘s an invaluable tool for model development.
-
Optimize data loading with tools like tf.data and prefetching. Data ETL is often the biggest performance bottleneck, especially with large datasets.
-
Take advantage of pretrained models from TensorFlow Hub or Keras applications for transfer learning. Don‘t reinvent the wheel!
-
Set random seeds for reproducibility. Nothing is more frustrating than inconsistent performance due to uncontrolled randomness.
-
Profile and optimize models for inference speed when deploying to production. Use tools like TF Lite or TensorFlow Serving to get the best performance on mobile/edge devices and web services.
Hopefully this in-depth overview gives you a solid understanding of the TensorFlow development landscape as it stands in 2024. The field is constantly evolving, but understanding the core abstractions of Sequential models, Functional DAGs, and subclassed models will serve you well no matter what the future holds.
