Supervised Learning
Linear regression
Predict a number as a weighted sum of features: y = w1*x1 + ... + b. Training minimises the mean squared error (MSE).
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
model = LinearRegression().fit(X_train, y_train)
pred = model.predict(X_test)
print("RMSE:", mean_squared_error(y_test, pred) ** 0.5)
print("R2:", r2_score(y_test, pred))
Logistic regression
Despite the name it is a classifier: it models the probability of a class with the sigmoid function and predicts the class when that probability crosses a threshold.
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
print(clf.predict_proba(X_test[:3]))
Other common models
| Model | Idea | Notes |
|---|---|---|
| k-NN | Vote of the k nearest points | Needs scaling; slow at prediction |
| Decision tree | Sequence of if/else splits | Interpretable; overfits easily |
| Random forest | Many trees on random subsets, averaged | Strong default, robust |
| Gradient boosting (XGBoost, LightGBM) | Trees built sequentially to fix errors | Often best on tabular data |
| SVM | Maximum-margin boundary | Good for small to medium data |
| Naive Bayes | Bayes theorem with independence assumption | Fast; good for text baselines |
Overfitting and regularisation
- L1 (lasso) shrinks some weights to zero (feature selection).
- L2 (ridge) shrinks weights smoothly.
- Limit tree depth, use more data, early stopping, dropout (neural networks).
Cross-validation and tuning
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, cross_val_score
rf = RandomForestClassifier(random_state=0)
print(cross_val_score(rf, X, y, cv=5).mean())
grid = GridSearchCV(rf, {"n_estimators": [100, 300], "max_depth": [None, 8]}, cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_)
NOTE
Tune on validation folds, then report the final score once on the untouched test set.
Loading Comments...