Developing a Machine Learning Model

Building a model is the easy part. You have written the cost function, the gradients and the network; you train it, and the predictions are not good enough. Now what? You could collect more data, add features, remove features, engineer polynomial terms, raise the regularization, lower it, or make the network bigger — each of which costs days or weeks, and most of which will do nothing. This note is about how to tell, before spending the time, which of them is worth trying. It is the difference between six months of guessing and two weeks of diagnosis.

In one line. Split the data three ways, compare $J_{\text{train}}$ and $J_{\text{cv}}$ against a baseline of what is achievable, and let the two gaps tell you whether you have a bias problem or a variance problem — because the fixes for one are useless for the other.

Evaluating a model

The train/test split

The cost after training tells you almost nothing. A model with enough parameters can drive $J$ to zero by memorising the training set — the wild degree-nine fit in linear-regression.md did exactly that — while being useless on anything it has not seen. What matters is generalization: performance on data that took no part in fitting the parameters.

So hold some data back. Split the examples, typically $70\%$ / $30\%$, fit the parameters on the training portion only, and evaluate on the untouched test portion. Write $m_{\text{train}}$ and $m_{\text{test}}$ for the two sizes.

Test error for regression and classification

The two errors are the same cost you already minimise, evaluated on the two different sets. For regression with squared error:

For classification, the same idea with the logistic loss:

For classification there is a second, more readable option: the fraction misclassified. Threshold each prediction at $0.5$ and count how often $\hat{y} \neq y$. That number is the one worth reporting to a human — "$5\%$ of test emails were sorted wrongly" means something in a way that a log loss of $0.18$ does not.

The evaluation cost carries no regularization

One detail that is easy to get wrong. The cost being minimised during training includes the regularization term; the errors being reported do not:

The penalty is a device for controlling overfitting, not part of the model's error. Including it in $J_{\text{test}}$ would mean two models with different $\lambda$ could not be compared at all, since each would be scored on a different quantity.

Model selection and the cross-validation set

Choosing with the test set, and why it breaks

Suppose you are choosing the degree $d$ of a polynomial model. The obvious procedure: fit all ten candidates on the training set, evaluate each on the test set, take the winner, and report its test error as the estimate of generalization performance.

The last step is where it fails. Say $d = 5$ wins with $J_{\text{test}}\left(w^{\langle 5\rangle}, b^{\langle 5\rangle}\right)$. That number is now an optimistic estimate, because $d$ was itself chosen by looking at the test set. The degree is an extra parameter, fitted to the test data by the act of picking the smallest number — exactly as $w$ and $b$ were fitted to the training data. Reporting it makes the same mistake as reporting $J_{\text{train}}$: the score comes from data the choice was tuned on.

The three-way split

The fix is a third split. Divide the data into training, cross-validation and test sets — a common ratio is $60/20/20$ — and give each one job. The cross-validation set is also called the validation or dev set.

The correct procedure

1. Train every candidate on the training set, giving $w^{\langle d\rangle}, b^{\langle d\rangle}$ for each.
2. Choose the candidate with the lowest $J_{\text{cv}}$.
3. Report $J_{\text{test}}$ for that one candidate — a set no decision has touched, so the number is an honest estimate of generalization error.

Nothing about this is specific to polynomial degree. The same three steps choose a neural network architecture — train a small, a medium and a large network, pick by $J_{\text{cv}}$, report $J_{\text{test}}$ — or a regularization strength, or a feature set. Any decision you make by looking at a dataset must be made on the cross-validation set, or the test set stops being a fair test.

Bias and variance

Three fits, and their signature

Fit a straight line, a quadratic and a degree-four polynomial to the same curved data and you get the three canonical outcomes — and, crucially, each leaves a distinct fingerprint in the two errors:

The single most useful sentence in this note: high bias shows as a large $J_{\text{train}}$; high variance shows as a large gap between $J_{\text{train}}$ and $J_{\text{cv}}$.

The two curves against model complexity

Sweep the degree and plot both errors together, and the whole picture appears at once.

Read the two ends. On the left, both curves are high and close together: the model cannot fit even the data it was given, and giving it more data would not change that. On the right, the training curve is at the floor while the validation curve has left the chart: the model fits its training set perfectly and has learned nothing transferable.

Both at once

High bias and high variance are not opposite ends of one dial — a model can have both. It shows as a training error that is already well above the baseline and a validation error well above the training error. In practice it happens when part of the input space is overfitted and part is underfitted, and it is more common with neural networks than with polynomials.

Regularization and bias/variance

What $\lambda$ does to the fit

The regularization strength $\lambda$ is a second, continuous, control on the same tradeoff, and it runs in the opposite direction to model complexity:

• $\lambda$ very large (say $10{,}000$): the penalty dominates, so minimising $J$ drives every $w_j \to 0$ and the model collapses to $f_{\vec{w},b}(\vec{x}) \approx b$ — a flat line through the mean. Maximum bias, complete underfit.
• $\lambda = 0$: no penalty at all, and a flexible model overfits freely. Maximum variance.
• Somewhere in between: the fit that tracks the trend and ignores the noise.

How $\lambda$ pushes the weights down

The mechanism is visible in one line of calculus. Differentiating the regularized cost of Equation 3 gives an extra term in every weight's gradient:

The weights therefore settle where the pull from the data balances the constant shrinkage from the penalty. Features that genuinely help keep their weight; features that only fit noise cannot justify theirs and are pulled to nothing.

Choosing $\lambda$ by cross-validation

$\lambda$ is chosen exactly like the degree was, by the procedure of §2.3. The standard search is a doubling ladder — $0$, $0.01$, $0.02$, $0.04$, $0.08$, and so on up to about $10$ — because what matters is the order of magnitude, not the third decimal place. Train each, take the lowest $J_{\text{cv}}$, report $J_{\text{test}}$ for the winner.

Put Figure 1 and Figure 2 side by side and they are the same picture reflected. Increasing the degree and decreasing $\lambda$ both move you from bias to variance; the validation curve is U-shaped in both, and its minimum is the model you want.

Establishing a baseline

Is $J_{\text{train}} = 10.8\%$ high or low? The question has no answer on its own. For recognising printed digits it would be a disaster; for transcribing noisy telephone speech it might be excellent. Before either error means anything you need a baseline: the error rate that is plausibly achievable on this problem. Reasonable sources are human performance on the same task, a competing algorithm's published number, or prior experience.

With a baseline, the two comparisons that matter become gaps rather than absolute levels:

Three sets of numbers, all with the same baseline of $10.6\%$, show how completely the diagnosis can change while the raw errors barely move:

Case A is the one people misread most often. A training error of $10.8\%$ looks bad in isolation — but the baseline says $10.6\%$ is the most anyone can do, so the model is essentially perfect on the training distribution and the entire problem is variance.

Learning curves

Bias and variance can also be diagnosed by varying the amount of data rather than the model. Train the same model on subsets of increasing size and plot both errors against $m_{\text{train}}$.

The shape, and why it has that shape

Two things happen as the training set grows, and they run in opposite directions:

• $J_{\text{cv}}$ falls. More data pins the parameters down better, so the model generalises better. This is the expected half.
• $J_{\text{train}}$ rises. This surprises people, and it should not: fitting four points perfectly is easy, fitting four hundred with the same model is not. The training error starts near zero and climbs as the model runs out of capacity to accommodate every example.

The two curves therefore approach each other from opposite sides and flatten. Where they flatten is the diagnosis.

High bias against high variance

That gives the most valuable prediction in this note, and it is worth stating baldly because it saves the most time:

If a model has high bias, more training data will not help. The curves have already flattened; adding examples moves them along a plateau. You need a more capable model — more features, polynomial terms, a bigger network, less regularization. If instead the model has high variance, the gap is still closing and more data is likely the single most effective thing you can do.

Plotting learning curves is expensive — you have to train the model many times over — but compared with six months spent collecting data that could never have helped, it is cheap.

Deciding what to try next

Now the list from the opening paragraph can be sorted. Every standard fix acts on one of the two problems, and applying a variance fix to a bias problem does nothing at all:

Three of the six increase model flexibility and three decrease it. That is the whole table: bias means not flexible enough, variance means too flexible, and every remedy is a move along that one axis.

Neural networks and the tradeoff

The recipe

Classically, bias and variance were a genuine dilemma: reducing one increased the other, and the job was to find the balance. Large neural networks changed that, because a big enough network is a low-bias machine — given enough units it can fit almost any training set you hand it. That turns the balancing act into a straightforward loop with two questions:

The loop has two practical limits. A bigger network eventually becomes computationally expensive — this is where GPUs enter the story — and "more data" is sometimes simply unavailable, which is what §11 is about.

Bigger is almost never worse, if you regularize

The obvious objection is that a huge network should overfit. It usually does not, provided the regularization is chosen appropriately: a larger network with a well-chosen $\lambda$ will usually do as well as or better than a smaller one. The cost is compute time, not accuracy. The regularized cost sums the penalty over every weight in the network:

In code the change is one argument per layer:

layer_1 = Dense(units=25, activation="relu",    kernel_regularizer=L2(0.01))
layer_2 = Dense(units=15, activation="relu",    kernel_regularizer=L2(0.01))
layer_3 = Dense(units=1,  activation="sigmoid", kernel_regularizer=L2(0.01))
model   = Sequential([layer_1, layer_2, layer_3])

The iterative loop of development

Zooming out from the model to the project, the same shape appears at a larger scale. Machine learning development is a loop, not a pipeline, and you go round it many times:

1. Choose an architecture — the model, the features, the data, the hyperparameters.
2. Train the model.
3. Run diagnostics — bias, variance, and error analysis.
4. Go back to step 1 with what you learned.

Take a spam classifier as the running example. The input $\vec{x}$ is a list of the top $10{,}000$ words, with $x_j$ recording whether word $j$ appears in the email; $y$ is $1$ for spam. The first round takes an afternoon. What the loop then produces is a sequence of decisions — add features for email routing, build a spelling normaliser, get more data of a specific kind — each one justified by a diagnostic rather than a guess. The two diagnostics that drive it are bias/variance and the subject of the next section.

Error analysis

Bias and variance tell you what kind of problem you have. Error analysis tells you what the problem is about, and it is nothing more than reading the mistakes:

1. Take the cross-validation examples the model got wrong.
2. If there are more than about a hundred, sample a hundred — the point is to find the big categories, and a sample of a hundred finds them as well as a thousand would.
3. Read them, and sort them into common themes by hand.
4. Count each theme.

For the spam classifier, a hundred misclassified emails might come back like this:

The deliberate-misspellings row is the point of the exercise. Building a spell-normaliser is a fun, tractable, obviously-useful-sounding project — and it could fix at most $3$ of $100$ errors. Pharma and phishing together account for $39$. Error analysis is mostly a tool for deciding what not to work on.

Its limitation is honest and worth knowing: it works best where a human can look at an example and say what the right answer was. For spam, a person can read the email and tell. For predicting which advertisement someone will click, no human can look at a failure and say what went wrong — and there, error analysis has much less to offer.

Adding data

When the diagnosis is high variance, more data helps — but "collect more data" is often the slowest, most expensive step in the project. There are cheaper routes to the same place.

Data augmentation

Augmentation modifies an existing training example to create a new one, with the label unchanged. For an image of the letter A, all of these are still an A: rotated, enlarged or shrunk, warped, drawn in a different colour or stroke width, partially occluded by background lines. One labelled example becomes ten, at no labelling cost.

The same idea works beyond images. For speech recognition, take one recording of "what is today's weather?" and overlay crowd noise, car noise, or the degradation of a bad phone connection — one clip becomes four, each with the same transcript.

The rule that makes it work. The distortion must be representative of the noise the model will actually meet at test time. Crowd noise helps because deployed audio really does have crowd noise. Adding purely random, meaningless per-pixel noise usually does not help, because nothing in the real world produces it — you have manufactured examples of a problem the model will never face, and spent capacity learning to ignore them.

Synthetic data

Augmentation modifies real examples; synthesis creates them from nothing. The classic case is photo OCR: instead of cropping and labelling letters from real photographs, generate them — take the fonts already on your computer, render text in various colours, sizes and distortions, and you have a large training set whose labels are known by construction.

The tradeoff is coverage. Real data contains oddities nobody would think to generate; synthetic data contains exactly what you told it to. It is at its best when the input is something a computer can plausibly render — text, code, structured audio — and at its weakest when the world is messier than your generator.

Data-centric development

Both techniques are instances of a broader shift in emphasis:

For decades the effort went into the algorithm, with the dataset held fixed — that is what a research benchmark encourages. In applied work the algorithms are now largely commodities, well implemented in libraries, and the leverage has moved to systematically improving the data: augmenting it, synthesising it, relabelling the noisy parts, and collecting more of the specific category error analysis pointed at.

Transfer learning

The last option applies when you have very little data and no way to make more. Transfer learning uses data from a different task:

1. Supervised pre-training. Take a network trained on a large dataset of the same input type — say a million images across a thousand classes of animals, vehicles and people. You rarely do this yourself; you download the parameters.
2. Fine-tuning. Replace the output layer with one shaped for your task — ten units for handwritten digits, say — and continue training on your own, much smaller dataset.

Why it works is the interesting part, and it is the same fact the deep-network reference makes about feature hierarchies. The early layers of an image network learn generic structure — the first detects edges, the next corners, the next curves and basic shapes — and none of that is specific to the task. Edges are edges whether the image contains a cat or a handwritten seven. Only the last layers are specialised, so only they need replacing. The one restriction: the input type must match. An image network transfers to images, an audio network to audio; neither transfers to the other.

The full cycle of a project

Everything above sits inside a still larger loop, which is what a real project looks like end to end:

The last box hides more engineering than it looks. Deploying typically means putting the model behind an inference server: the application makes an API call with the input, the server returns the prediction. Around that sits the work of making it reliable — efficient predictions, scaling with load, logging inputs and outputs, monitoring for the day the live data stops resembling the training data, and a process for pushing model updates. That discipline has a name, MLOps, and on a production system it is a larger body of work than training the model was.

Skewed datasets: precision and recall

Why accuracy lies

Consider a classifier for a rare disease, $y = 1$ if present. You train it and get $1\%$ error on the test set — $99\%$ of diagnoses correct. That sounds excellent until you learn that only $0.5\%$ of patients have the disease, because then this program does better:

def predict(x):
    return 0          # "nobody has the disease"

It is right $99.5\%$ of the time. It has $0.5\%$ error, beating your model, and it is completely worthless. On a skewed dataset — one class far rarer than the other — accuracy is dominated by the majority class and stops measuring anything you care about.

The same effect on the real classifier used for the figures below: predicting "negative" for every one of $4000$ patients scores $97.25\%$ accuracy, while the trained model scores $97.98\%$. Judged on accuracy the two are nearly identical, and one of them detects no cases at all.

The confusion matrix

The fix is to stop compressing the outcome into one number too early. There are four outcomes, not two, and they are laid out in the confusion matrix:

Precision and recall

Two ratios pull the useful information out of that table, and the difference between them is entirely in the denominator:

In words, and it is worth learning them as sentences rather than formulas:

• Precision — of everyone I flagged, what fraction really had it? Here, of the $20$ people told they were ill, $75\%$ were. Precision is what you care about when a false positive is expensive or harmful: unnecessary surgery, a legitimate email deleted as spam.
• Recall — of everyone who really had it, what fraction did I catch? Here, of the $25$ ill people, only $60\%$ were found. Recall is what you care about when a false negative is dangerous: a missed diagnosis, an undetected fraud.

Notice that the "predict $0$ always" program has recall exactly $0$, and its precision is undefined because it never predicts a positive at all. The two metrics catch the cheat instantly where accuracy did not.

Trading them off

You are not stuck with whatever precision and recall the model happens to give, because the threshold is yours to choose. Logistic regression outputs a probability; the decision to call it positive is a separate step:

• Raise the threshold (say to $0.9$): predict positive only when very confident. Fewer false positives, so precision rises; more missed cases, so recall falls.
• Lower it (say to $0.1$): predict positive when in any doubt. Catch nearly everyone, so recall rises; many false alarms, so precision falls.

The $F_1$ score

Two numbers are awkward when you have to rank candidates. The obvious repair — average them — fails badly, and the failure is instructive:

The $F_1$ score is the harmonic mean of precision and recall, which is the average that refuses to be gamed:

The mechanism is worth seeing. If $R \to 0$ then $1/R \to \infty$, the denominator explodes, and $F_1 \to 0$ no matter how good $P$ is. The harmonic mean is close to the smaller number, where the arithmetic mean sits halfway between. So $F_1$ is high only when precision and recall are both high, and it cannot be raised by sacrificing one for the other — which is exactly the property you want from a metric on a skewed problem.

Where this goes next

The diagnostic toolkit here is model-agnostic on purpose. Train/cv/test, bias and variance, learning curves, error analysis and $F_1$ apply just as well to a decision tree or a gradient boosted ensemble as to the networks they were introduced with — and the next note is about exactly those: decision trees, which split on features rather than weighting them, are trained by maximising information gain rather than by gradient descent, and then combine into random forests and XGBoost.

Backwards from here: multiclass-softmax.md builds the classifier this note evaluates, logistic-regression.md and linear-regression.md hold the models and the regularization underneath it, and neural-networks-reference.md covers the architectures the recipe in §8 makes bigger.