Neural networks

A whole layer in one line

Writing out $w_1 x_1 + w_2 x_2 + \dots$ for every single neuron gets old fast. So we stack the inputs into a vector and every weight into a matrix, and let the maths run a full layer at once:

Here $\mathbf{x}$ is the column of inputs, $W$ is a grid holding every weight (one row per neuron), $\mathbf{b}$ is a small nudge for each neuron (its bias), and $\sigma$ is applied to each result. One tidy line does the work of the whole layer. It is also why graphics cards, built for exactly this kind of matrix multiply, train networks so fast.

Sigmoid or ReLU?

The squash back in Lesson 2 was the sigmoid. It is perfect for a final answer between 0 and 1, but buried inside deep networks it makes learning crawl. So hidden layers today almost always use ReLU, which could not be simpler:

Keep the number if it is positive, otherwise send zero. That one little bend is enough to let a stack of neurons curve around complicated data, it is dirt cheap to compute, and it dodges the "signal fades to nothing" problem that deep sigmoid stacks suffer from.

How a network actually learns

A fresh network starts with random weights, so its first guesses are useless. Training fixes that with a loop you already half-know from Lesson 1:

1. Predict: push one training match through the layers and see what comes out.
2. Measure: compare the guess to the real result with a loss, a single number for how wrong it was.
3. Nudge: shift every weight a hair in the direction that would have made the loss smaller.
4. Repeat across the whole dataset, thousands of times, until the loss stops falling.

Each nudge follows one rule, applied to every weight, millions of times over:

In plain words: move each weight a little in the direction that lowers the loss $L$, and the step size $\eta$ is the learning rate. Working out that direction layer by layer, backwards from the answer, is backpropagation. Taking the step again and again is gradient descent. That is the engine under every neural network, from this tiny one to the giants behind chatbots.

The same network in real code

You will almost never wire this by hand. Here is our win-predictor as scikit-learn would build it: one hidden layer of four neurons, ReLU inside, trained in a single call. Press Run.

from sklearn.neural_network import MLPClassifier
import numpy as np

# each row: [ranking gap, recent form, playing at home?]  scaled 0-1
X = np.array([[0.9, 0.7, 1], [0.8, 0.6, 0], [0.2, 0.3, 1],
              [0.1, 0.2, 0], [0.6, 0.8, 1], [0.3, 0.4, 0]])
y = np.array([1, 1, 0, 0, 1, 0])          # 1 = won, 0 = did not

# one hidden layer of 4 neurons, ReLU inside
net = MLPClassifier(hidden_layer_sizes=(4,), activation="relu",
                    max_iter=2000, random_state=1)
net.fit(X, y)                          # gradient descent runs here

new = [[0.85, 0.75, 1]]                # strong side, in form, at home
print("Win probability:", net.predict_proba(new)[0][1].round(3))
print("Call:", "WIN" if net.predict(new)[0] == 1 else "NOT A WIN")

Win probability: 0.938
Call: WIN

Same shape as the lab above: features in, one hidden layer, a probability out. The library hides the matrix maths and the gradient descent, but it is doing exactly what we just walked through.