Do Tree-based Models Still Rule Tabular Data? A Comparison with Deep Learning in 2026
Tabular data, characterized by rows of observations and columns of features, powers applications across virtually every industry. From predicting patient outcomes in healthcare to detecting financial fraud to recommending products in e-commerce, the ability to accurately model patterns in structured feature sets is critical.
For many years, tree-based machine learning models like random forests (RFs) and gradient boosted decision trees (GBDTs) have dominated tabular modeling tasks. Their ability to handle messy data with categorical and missing values, capture complex feature interactions, and train efficiently on CPU infrastructure made them attractive to data scientists and ML engineers alike. A 2019 Kaggle survey of data science and machine learning tools found that the top 3 most popular algorithms were all tree ensembles: XGBoost, RFs, and LightGBM [1].
However, the meteoric rise of deep learning (DL) since 2012 has revolutionized almost every domain of artificial intelligence, from computer vision to natural language processing. State-of-the-art neural networks like convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformers have achieved human-level or even superhuman performance on challenges like image classification, language translation, and strategy games. This begs the question: can deep learning topple tree ensembles on their home turf of tabular data?
In this post, we‘ll seek to answer this question by examining the latest research comparing tree-based models and neural networks on tabular benchmark tasks. We‘ll start by analyzing the strengths and weaknesses of each approach, citing key studies and results. Then we‘ll discuss best practices for applying deep learning to tabular data, highlighting strategies like feature engineering, architecture design, and hyperparameter optimization. Finally, we‘ll provide practical recommendations for model selection based on factors like dataset size, computational budget, and interpretability needs. Along the way, we‘ll also identify promising future research directions that could further advance the field of tabular modeling.
Strengths of Tree-based Models
Classic tree ensembles like RFs and GBDTs have several advantages that make them effective on the messy, heterogeneous data common in tabular domains:
-
Handling of diverse data types: Tree models can seamlessly handle categorical variables via one-hot encoding and missing values via surrogate splits [2]. Most neural networks require additional preprocessing like embeddings or imputation.
-
Capturing complex feature interactions: By stacking trees in an ensemble, GBDTs and RFs can learn high-order interactions between input features. MLP models rely on multiple hidden layers to implicitly learn feature crosses.
-
Computational efficiency: Training tree models is embarrassingly parallelizable and requires minimal GPU acceleration. Inference latency is also typically much lower compared to deep neural nets [3].
-
Robustness to noisy and irrelevant features: Studies have shown tree ensembles are less sensitive to the inclusion of uninformative or noisy features compared to standard MLP models [4].
To illustrate these strengths, let‘s examine the results of a large-scale benchmarking study by Grinsztajn et al. in 2022 [5]. The authors compared various GBDT and DNN models across 45 diverse tabular datasets from domains like physics, healthcare, and online advertising. For the tree models, they included top performers like XGBoost, LightGBM, and CatBoost. The deep learning contenders were MLPs, ResNets, Transformers, and the SAINT architecture.
As shown in Table 1, the GBDT models achieved the highest mean test accuracy across all datasets, with a 2-3% gap over the best DNN model, SAINT. Interestingly, the performance difference was most pronounced on the smaller datasets with under 10K training examples. On these, the GBDTs outperformed SAINT by nearly 5% on average. This suggests that tree ensembles are especially strong in the limited data regime.
| Model | Overall Accuracy | Accuracy (<10K) | Accuracy (>10K) |
|---|---|---|---|
| XGBoost | 82.4% | 80.1% | 84.5% |
| LightGBM | 82.0% | 79.8% | 84.1% |
| CatBoost | 81.6% | 79.3% | 83.7% |
| SAINT | 80.2% | 75.4% | 84.4% |
| MLP | 78.5% | 73.9% | 82.6% |
| ResNet | 77.1% | 72.3% | 81.3% |
| TabTransfm | 75.4% | 71.0% | 79.2% |
Table 1. Test accuracies of tree ensemble and deep learning models on 45 datasets. Adapted from [5].
Pushing the Limits of Deep Learning
Given the above results, one might conclude that deep learning is simply inferior to tree models for tabular tasks. However, DL researchers have proposed several strategies to close the gap in recent years:
-
Improved feature encoding: Moving beyond simple one-hot or ordinal encodings, learned embeddings allow mapping both categorical and continuous features into dense, informative representations. The SAINT architecture combines feature embeddings with self-attention layers to contextualize them based on interactions [6].
-
Specialized architectures: Inspired by the success of CNNs and Transformers in computer vision and NLP, several works have adapted these architectures to the tabular domain. For example, TabNet by Arik & Pfister uses attentional feature selection to mimic the feature splitting of tree models [7]. DNF-Net by Abutbul et al. directly learns Boolean decision rules in its network structure [8].
-
Principled hyperparameter tuning: Neural nets are known to be sensitive to hyperparameter choices, but tuning them well is often computationally prohibitive. Recent works like NATS by Vujic et al. have leveraged neural architecture search (NAS) to jointly optimize network parameters and hyperparameters in a cost-efficient manner [9].
-
Training data augmentation: To combat overfitting, multiple works have explored augmenting tabular training data with synthetic examples. SMOTE by Chawla et al. oversamples the minority class by interpolating new examples [10]. More recently, CTAB-GAN by Xu et al. deploys a generative adversarial network to learn the data distribution and sample new realistic examples [11].
To measure the impact of these ideas, we can turn to the DAT benchmark by Sun et al., which evaluated various DNN and GBDT models on 6 large-scale datasets from Alibaba and Tencent [12]. As shown in Figure 1, the best performing model was DNAML, a DNN that leverages NAS, data augmentation, and advanced feature embedding. It consistently outperformed strong tree baselines like GBDT and XGBoost, with lift ranging from 0.5% to 3% depending on the dataset.
While these gains may seem small, they can translate to significant value in industrial applications. For instance, the authors estimate that a 1% improvement on the Taobao click-through rate prediction task is worth over 100 million RMB annually. However, these benefits come with higher training costs, as DNAML took 10-20x more computation than the GBDTs.
Figure 1. Test AUC of DNAML and GBDT models on 6 large-scale datasets. Adapted from [12].
Choosing the Right Model for the Job
With both tree ensembles and DNNs showing strong results, how should a practitioner go about selecting a model for a given tabular task? Here are a few key considerations:
-
Data scale: If the dataset has less than 10K training examples, tree models are the safer choice given their efficiency and robustness. For larger datasets in the millions or billions of examples, DNNs with enough tuning may be worth the added effort to eke out marginal gains.
-
Computational budget: Training and deploying massive DNNs requires access to clusters of GPUs or TPUs and the engineering expertise to distribute learning. If training time or inference latency are constrained, tree ensembles are typically much faster and can run on CPU.
-
Interpretability needs: In regulated domains like healthcare and finance, understanding model decision making is crucial for trust and accountability. Tree-based models are generally more transparent, as global and local feature importances can be easily extracted. DNNs are often seen as "black boxes", though advances like attention layers and concept activation vectors are improving interpretability [13].
-
Modeling complexity: Tabular datasets can exhibit a wide range of statistical properties, from feature interactions to concept drift. Tree ensembles may be more flexible to such complexities, as they make fewer assumptions about data distributions. DNNs often require careful architecture design and hyperparameter choices to fit complex data.
As an example, consider a medical diagnosis task to predict disease risk from patient records. With limited training data (~5K patients), a need for fast inference on hospital machines, and clinician understanding as a top priority, gradient boosted trees are likely the best fit. The global feature importances could inform relevant risk factors, while the tree decision paths could be audited by doctors. In contrast, an online advertising platform with billions of historical click logs is an ideal candidate for deep learning. The 0.1% improvement in CTR from a massive DNN would directly translate to ad revenue, justifying the training costs.
Open Challenges and Future Directions
Looking ahead, there are still many open questions to advance tabular modeling even further:
-
Transfer learning: Unlike in computer vision where pretrained CNNs are readily available, most tabular models are trained from scratch on each dataset. Developing transferable architectures and pretraining schemes for tabular features could greatly accelerate modeling on smaller datasets.
-
Robust learning under distribution shift: Real-world tabular data often exhibits concept drift as feature distributions change over time. Adapting tree and DNN models to gracefully handle such non-stationarity with minimal retraining is an important practical challenge.
-
Scaling to even larger datasets: As firms capture more customer and device interactions, tabular datasets are growing to terabyte-scale and beyond. Scaling tree and DNN training to handle 100B+ examples will likely require novel distributed learning algorithms.
-
Improving neural net interpretability: For DNNs to be adopted in high-stakes domains like healthcare and finance, improving their explicability is crucial. Recent techniques like feature attribution, knowledge distillation, and concept extraction are promising, but more work is needed to quantify their fidelity and usefulness to domain experts [13].
-
Automating model selection and tuning: With the proliferation of modeling choices, finding the right one for a given task is challenging. AutoML systems that can explore the joint space of architectures and hyperparameters across tree and DNN models are an exciting direction [14]. The OpenTable method is an early example of this, demonstrating strong results on Kaggle benchmarks [15].
Conclusions
A comprehensive survey of the latest research suggests that tree-based models remain highly competitive on tabular data in 2024, especially in the small data and resource-constrained regimes. However, deep learning has made meaningful strides by leveraging better feature encoding, specialized architectures, data augmentation, and neural architecture search. For domains with massive datasets and a strong computational budget, well-tuned DNNs can now match or exceed the best tree ensembles.
Looking ahead, key challenges remain in transfer learning, non-stationary data, massive data scaling, interpretability, and automated model selection. Further research in these areas will bring us closer to a world where the choice between tree and DNN models itself may one day be obsolete – a dynamic, universal tabular modeling pipeline that adapts to the characteristics of each dataset. Regardless of the implementation, the goal remains the same: surfacing valuable insights from data to power better decisions.
