Preparing features: encoding, scaling, missing values and avoiding leakage

Most of the time in a machine learning job goes into preparing the data, not choosing the model. Models need numbers with no gaps, on comparable scales, and with no information that would not exist at prediction time.

Encoding categories

Text categories must become numbers. One-hot encoding creates one 0/1 column per category. Example 1: a Colour column with red, green and blue becomes three columns; a red row is 1, 0, 0. Use it when categories have no order. Ordinal encoding maps categories to integers and is right only when order means something, such as small = 1, medium = 2, large = 3. Mapping city names to 1, 2, 3 tells a linear model that city 3 is "three times" city 1, which is meaningless.

Scaling and missing values

Distance-based and gradient-based models (k-NN, k-means, logistic regression, neural networks) need features on similar scales. Standardisation subtracts the mean and divides by the standard deviation, so each feature has mean 0 and standard deviation 1. Min-max scaling squeezes values into 0 to 1. Tree-based models do not need scaling.

Example 2: ages from 20 to 60 and incomes from 20,000 to 2,00,000. Without scaling, every distance in k-NN is dominated by income; age barely matters. After standardisation both contribute.

For missing values the options are: drop rows (only when few), fill with the median for numbers or the most common value for categories, or add a flag column such as was_missing so the model can learn that missingness itself carries information. Example 3: a loan dataset where income is blank more often for rejected applications. Filling with the median and discarding the fact that it was blank throws away a useful signal.

Leakage and the fit-then-transform rule

Leakage is any information in the training features that would not be available when the model is used. Example 4: predicting whether a customer will cancel, using a column refund_issued that is only filled after they cancel. Training accuracy looks excellent; in production the column is always blank.

A subtler form: computing the scaling mean, the imputation median or the one-hot categories on the full dataset before splitting. The test set has then influenced the preprocessing. The rule is fit on training data, transform both. In scikit-learn terms, call scaler.fit(X_train), then scaler.transform(X_train) and scaler.transform(X_test), and never fit on the test set. A Pipeline enforces this.

Common mistakes and a practice habit

• Ordinal encoding categories that have no order.
• Scaling before splitting, or re-fitting the scaler on the test set.

Take a small dataset and build a scikit-learn Pipeline with an imputer, an encoder, a scaler and a model. Evaluate with cross-validation. Then deliberately leak a column derived from the target and watch the score jump. Once you have seen leakage happen, you will recognise it in any dataset.