Splitting data, overfitting and the bias-variance trade-off

A model is only useful if it works on data it has not seen. Most machine learning screening questions test whether you can set up an honest evaluation and diagnose what goes wrong. This applies to supervised learning, where training data includes the answer (regression for numbers, classification for categories), and the same discipline applies to tuning unsupervised methods such as clustering.

Train, validation and test sets

A common setup splits data into a training set to fit the model, a validation set to compare settings, and a test set used once at the end. Common splits are around 70/15/15 or 80/20, but the principle matters more than the numbers: the test set must not influence any decision.

A frequent hidden mistake is data leakage. Fitting a scaler or filling missing values using the whole dataset before splitting lets test information reach the model. Always split first, then fit preprocessing on the training set only.

Overfitting and underfitting

• Overfitting (high variance): training score is high, test score much lower. The model memorised noise. Fixes: more data, regularisation, a simpler model, fewer features.
• Underfitting (high bias): both scores are poor and close together. The model is too simple. Fixes: more flexible model, better features, less regularisation.

Model complexity moves you along this trade-off. A k-NN model with k = 1 or a very deep decision tree tends to overfit; a straight line fitted to a curved pattern underfits.

Cross-validation

A single split can be lucky or unlucky. In k-fold cross-validation, the data is cut into k parts. The model trains k times, each time validating on a different part, and the scores are averaged.

5 folds: train on folds 2-5, validate on 1
         train on 1, 3-5, validate on 2
         ... and so on, 5 rounds in total

Cross-validation gives a more stable estimate and also shows how much scores vary between folds.

Checklist before you report a score

1. Was the test set untouched until the end?
2. Was preprocessing fitted on training data only?
3. Did you compare training and validation scores?
4. Is the score better than a simple baseline, such as always predicting the most common class?