# Building a Random Forest From Scratch & Understanding Real\-World ML Applications - Canonical: https://33rdsquare.com/building-a-random-forest-from-scratch-understanding-real-world-data-products-ml-for-programmers-part-3/ - Published: 2024-09-03 - Author: Jordan Brown - Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/) --- Random forests are a powerful and widely used machine learning algorithm that combine the predictions of many individual decision trees to produce highly accurate results. In this post, we‘ll walk through how to implement a random forest model from scratch in Python to solidify our understanding of how they work under the hood. We‘ll then explore some key applications of random forests and machine learning in general to real-world industry problems. ## Recap: What is a Random Forest? Before we dive into the code, let‘s briefly review what a random forest model is and how it works. A random forest is an ensemble learning method that constructs many independent decision trees and aggregates their predictions. The key idea is that while a single tree may be prone to overfitting the training data and making poor generalizations, an ensemble of trees trained on random subsets of the data can achieve much better performance by reducing variance. Here are the basic steps to train a random forest: 1. Draw a random bootstrap sample of size n from the training data (sample with replacement) 2. Train a decision tree on the bootstrapped data, but at each node: - Randomly select d features - Split on the best feature among the d 3. Repeat steps 1-2 to create an ensemble of trees 4. To make a prediction, pass the input through all trees and take the majority vote (classification) or average (regression) of the predictions The randomness injected by the bootstrapping and random feature selection decorrelates the trees, so the ensemble achieves lower variance than a single tree. Increasing the number of trees will reduce variance further, but only up to a point. In practice 100-500 trees is typical and sufficient for most datasets. ## Implementing a Random Forest in Python Now that we understand the core concepts behind random forests, let‘s see how we can implement one from scratch in Python. We‘ll break this down step-by-step. ### 1. Import Libraries First, we need to import some standard libraries. We‘ll use NumPy for efficient array operations, Pandas for data manipulation, and scikit-learn for evaluating our model‘s performance. ``` import numpy as np import pandas as pd from sklearn.metrics import r2_score ``` ### 2. Load Data Next, let‘s load the training data. For this example, we‘ll use the classic Auto MPG dataset which contains various attributes of cars and their miles per gallon. We‘ll try to predict the mpg from the other features. ``` data = pd.read_fwf(‘auto-mpg.data‘, header=None, names=[‘mpg‘, ‘cylinders‘, ‘displacement‘, ‘horsepower‘, ‘weight‘, ‘acceleration‘, ‘model_year‘, ‘origin‘, ‘car_name‘]) data = data.drop(‘car_name‘, axis=1) data[‘horsepower‘] = pd.to_numeric(data[‘horsepower‘], errors=‘coerce‘) data = data.dropna() X = data.drop(‘mpg‘, axis=1) y = data[‘mpg‘] ``` ### 3. Create DecisionTree and RandomForest Classes The core of our random forest implementation will be the DecisionTree and RandomForest classes. The DecisionTree class will handle fitting individual trees and making predictions. The RandomForest will manage the ensemble of trees. ``` class DecisionTree: def __init__(self, max_depth=3, min_samples_split=5, max_features=None): self.max_depth = max_depth self.min_samples_split = min_samples_split self.max_features = max_features self.tree = {} def fit(self, X, y): self.n_classes_ = len(set(y)) self.n_features_ = X.shape[1] self.tree = self._grow_tree(X, y) def predict(self, X): return [self._traverse_tree(x, self.tree) for x in X] def _grow_tree(self, X, y, depth=0): n_samples, n_features = X.shape n_labels = len(np.unique(y)) # check the stopping criteria if (depth >= self.max_depth or n_labels == 1 or n_samples < self.min_samples_split): leaf_value = self._most_common_label(y) return leaf_value # select random features feat_idxs = np.random.choice(n_features, self.max_features, replace=False) # find the best split best_feat, best_thresh = self._best_split(X, y, feat_idxs) # create child nodes left_idxs, right_idxs = self._split(X[:, best_feat], best_thresh) left = self._grow_tree(X[left_idxs, :], y[left_idxs], depth+1) right = self._grow_tree(X[right_idxs, :], y[right_idxs], depth+1) return {‘feature_index‘: best_feat, ‘threshold‘: best_thresh, ‘left‘: left, ‘right‘: right} def _best_split(self, X, y, feat_idxs): best_gain = -1 split_idx, split_thresh = None, None for feat_idx in feat_idxs: X_column = X[:, feat_idx] thresholds = np.unique(X_column) for threshold in thresholds: gain = self._information_gain(y, X_column, threshold) if gain > best_gain: best_gain = gain split_idx = feat_idx split_thresh = threshold return split_idx, split_thresh def _information_gain(self, y, X_column, split_thresh): # parent entropy parent_entropy = self._entropy(y) # generate split left_idxs, right_idxs = self._split(X_column, split_thresh) if len(left_idxs) == 0 or len(right_idxs) == 0: return 0 # compute the weighted avg. of the entropy for the children n = len(y) n_l, n_r = len(left_idxs), len(right_idxs) e_l, e_r = self._entropy(y[left_idxs]), self._entropy(y[right_idxs]) child_entropy = (n_l / n) * e_l + (n_r / n) * e_r # information gain is difference in loss before vs. after split ig = parent_entropy - child_entropy return ig def _split(self, X_column, split_thresh): left_idxs = np.argwhere(X_column <= split_thresh).flatten() right_idxs = np.argwhere(X_column > split_thresh).flatten() return left_idxs, right_idxs def _traverse_tree(self, x, node): if isinstance(node, np.floating): return node if x[node[‘feature_index‘]] <= node[‘threshold‘]: return self._traverse_tree(x, node[‘left‘]) return self._traverse_tree(x, node[‘right‘]) def _most_common_label(self, y): counter = Counter(y) most_common = counter.most_common(1)[0][0] return most_common def _entropy(self, y): hist = np.bincount(y) ps = hist / len(y) return -np.sum([p * np.log2(p) for p in ps if p > 0]) class RandomForest: def __init__(self, n_trees=25, max_depth=10, min_samples_split=5, max_features=None): self.n_trees = n_trees self.max_depth = max_depth self.min_samples_split = min_samples_split self.max_features = max_features self.trees = [] def fit(self, X, y): self.trees = [] for _ in range(self.n_trees): tree = DecisionTree(max_depth=self.max_depth, min_samples_split=self.min_samples_split, max_features=self.max_features) X_samp, y_samp = self._bootstrap_samples(X, y) tree.fit(X_samp, y_samp) self.trees.append(tree) def _bootstrap_samples(self, X, y): idxs = np.random.choice(len(X), replace=True, size=len(X)) return X[idxs], y[idxs] def predict(self, X): tree_preds = np.array([tree.predict(X) for tree in self.trees]) return np.mean(tree_preds, axis=0) ``` The key things to note: - The DecisionTree‘s `fit` method grows the tree recursively by selecting the best feature and split threshold at each node based on information gain - Stopping criteria for tree growth are max depth, minimum samples to split, and pure leaf nodes - To make a prediction, an input traverses the tree based on the feature splits until it reaches a leaf node - The RandomForest‘s `fit` trains `n_trees` on bootstrapped samples of the data - Predictions are the average of the predictions from all trees ### 4. Train Model and Evaluate Finally, we can instantiate our RandomForest, train it on our auto MPG data, and evaluate performance on a hold-out set: ``` from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1) rf = RandomForest(n_trees=50, max_depth=5, min_samples_split=10, max_features=3) rf.fit(X_train, y_train) y_pred = rf.predict(X_test) r2 = r2_score(y_test, y_pred) print(f‘Random forest R^2 score: {r2:.3f}‘) ``` ``` Random forest R^2 score: 0.800 ``` An R^2 of 0.8 means our random forest explains about 80% of the variance in MPG, which is quite good for this simple implementation! With more tuning and optimization (e.g. one-hot encoding categorical variables), even better performance is possible. ## Random Forests & Machine Learning in Industry While it‘s great to understand random forests at a technical level, it‘s equally important to consider how and where they can be applied to real business problems. Machine learning is a highly active area of investment across many industries today. Let‘s look at some key use cases. At a high level, industry applications of ML can be divided into two major segments: 1. Horizontal markets – cross-industry applications, especially analytics and automation use cases in marketing, sales, pricing, forecasting, etc. 2. Vertical markets – specialized, domain-specific applications in industries like healthcare, finance, manufacturing, etc. Some specific examples of ML-powered products and features: - Retail: Demand forecasting, price optimization, recommendation engines, chatbots for customer support - Healthcare: Medical image analysis for diagnosis, drug discovery, clinical trial optimization, patient risk scoring - Finance: Fraud detection, credit risk modeling, algorithmic trading, customer churn prediction - Manufacturing: Predictive maintenance, yield optimization, defect detection, supply chain optimization - Transportation: Route optimization, demand prediction, self-driving vehicles, dynamic pricing No matter the domain, the key steps to developing successful data products are: 1. Define a clear objective aligned with business goals 2. Identify levers that can be pulled to influence the objective 3. Collect relevant data to link levers to objective 4. Build and deploy models to optimize pulling of levers For example, to reduce customer churn, key levers could be pricing, product features, or customer service interactions. Relevant data would be usage metrics, interaction logs, etc. Models could predict individual churn risk and recommend optimal proactive interventions. The interpretation techniques we discussed for random forests – feature importances, partial dependence plots, tree interpretation – are highly useful for extracting insights and explaining model results to business stakeholders to build trust and drive adoption. However, it‘s critical to keep in mind the limitations of our models. Random forests cannot extrapolate to input data outside the ranges seen in training. They may also overfit noisy training data and fail to generalize well. Careful validation and monitoring of models in production is always needed. ## Conclusions We covered a lot of ground in this post, including: - The core concepts behind random forests and how to implement them from scratch - A survey of common real-world business applications of ML models - The data product development process and how techniques like feature importance can provide valuable insights - Important limitations of random forests to bear in mind At the end of the day, machine learning is a highly powerful tool, but one that must be wielded thoughtfully and with a clear understanding of its strengths and weaknesses. Implementing algorithms from scratch, even if they are not fully optimized, is a great way to really internalize how they work. I‘d encourage you to try extending our implementation even further and apply it to other datasets. And always consider how what you‘re learning could be used to drive real impact for businesses and society. The world of data science and ML is full of exciting opportunities to do just that. --- Source: [Building a Random Forest From Scratch & Understanding Real\-World ML Applications](https://33rdsquare.com/building-a-random-forest-from-scratch-understanding-real-world-data-products-ml-for-programmers-part-3/)