Math Foundations for ML

You do not need advanced mathematics to start, but three ideas explain most of machine learning: vectors and matrices, derivatives, and optimisation.

Vectors and matrices​

A data set with n rows and d features is a matrix X of shape (n, d). A model's weights are a vector w of length d.

import numpy as np

X = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])   # 3 samples, 2 features
w = np.array([0.5, -1.0])
b = 2.0

pred = X @ w + b          # matrix-vector product: one prediction per row
print(pred)               # [0.5 0.5 0.5]  (each row: 0.5*x1 - 1.0*x2 + 2.0)

Derivatives and gradients​

The derivative tells how a function changes when its input changes. For many inputs the gradient is the vector of partial derivatives and points in the direction of steepest increase.

For squared error L(w) = (w*x - y)^2 the gradient with respect to w is 2 * x * (w*x - y).

Gradient descent​

To minimise a loss, repeatedly step against the gradient.

import numpy as np

# fit y = w * x
x = np.array([1.0, 2.0, 3.0, 4.0])
y = np.array([3.0, 6.0, 9.0, 12.0])    # true w = 3

w, lr = 0.0, 0.01
for step in range(200):
    pred = w * x
    grad = (2 * x * (pred - y)).mean()  # d(MSE)/dw
    w -= lr * grad

print(round(w, 3))                      # about 3.0

Why it matters​

Training a neural network is the same loop on millions of parameters: compute the loss, compute gradients with backpropagation, update weights. See Unsupervised learning and neural networks.