Decision Trees and Ensemble Methods
Decision trees
A tree repeatedly splits the data on the feature and threshold that best separates the target.
| Split criterion | Used for | Idea |
|---|---|---|
| Gini impurity / entropy | Classification | Make child nodes as pure as possible |
| Variance / MSE reduction | Regression | Make child nodes have similar values |
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X, y)
print(export_text(tree, feature_names=load_iris().feature_names))
Deep trees overfit. Control with max_depth, min_samples_leaf or pruning.
Ensembles
Combining many weak or unstable models gives a strong, stable one.
| Method | How | Reduces |
|---|---|---|
| Bagging (random forest) | Train trees on bootstrap samples and random feature subsets; average | Variance |
| Boosting (gradient boosting, XGBoost, LightGBM) | Add trees sequentially, each fitting the previous errors | Bias |
| Stacking | A meta-model learns to combine base models | Both |
Random forest vs gradient boosting
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
from sklearn.model_selection import cross_val_score
rf = RandomForestClassifier(n_estimators=300, n_jobs=-1, random_state=0)
gb = HistGradientBoostingClassifier(learning_rate=0.1, max_depth=6, random_state=0)
for name, model in [("random forest", rf), ("gradient boosting", gb)]:
print(name, cross_val_score(model, X, y, cv=5).mean().round(3))
| Random forest | Gradient boosting | |
|---|---|---|
| Tuning effort | Low | Higher |
| Overfitting risk | Low | Moderate (use early stopping) |
| Accuracy on tabular data | Strong | Often best |
| Training | Parallel | Sequential |
Feature importance
rf.fit(X, y)
for name, imp in sorted(zip(load_iris().feature_names, rf.feature_importances_), key=lambda t: -t[1]):
print(f"{name:25s} {imp:.3f}")
NOTE
Impurity-based importances are biased towards high-cardinality features. For more reliable explanations use permutation importance or SHAP.