Missing values and data types: isna, fillna, dropna and conversions

Real data arrives with gaps and wrong types. A column of prices is text because one cell says N/A; a date column is text because the export used day-month-year. Until these are fixed, sums are wrong, sorts are alphabetical and filters miss rows. This lesson covers the small set of pandas tools that fix them, and the mistakes that look like they worked.

Finding missing values

pandas marks missing values as NaN (or NaT for dates). df.isna() returns True where a value is missing; df.isna().sum() gives a count per column, which is the first thing to run on any new file. df[df['email'].isna()] shows the rows with no email.

Example 1: a DataFrame with 1,000 rows where df.isna().sum() shows phone 120 and city 3. That tells you phone is often blank and city is almost complete, so you will treat them differently.

Dropping or filling

df.dropna() removes any row with at least one NaN, which is usually too aggressive. df.dropna(subset=['city']) removes only rows where city is missing. df.fillna({'phone': 'unknown', 'quantity': 0}) fills each column with its own value. For numeric columns, filling with the median is common: df['age'] = df['age'].fillna(df['age'].median()).

Example 2: a sales column with NaN in 5 of 100 rows. df['sales'].sum() skips NaN and returns the total of the 95. df['sales'].mean() divides by 95, not 100. If a missing value really means zero, fill it first; otherwise the mean describes the rows that had data, which may be what you want. Decide, then record it in a comment.

None of these change the DataFrame unless you assign the result back. Assigning back is clearer than inplace=True.

Converting types

df.dtypes shows each column's type. A numeric column showing object is text. pd.to_numeric(df['price'], errors='coerce') converts what it can and turns the rest into NaN, so "12.5" becomes 12.5 and "N/A" becomes NaN. Without errors='coerce' the first bad value raises an error, which is sometimes what you want so you see it.

Example 3: prices like "1,250" need the comma removed first: pd.to_numeric(df['price'].str.replace(',', ''), errors='coerce').

Dates: pd.to_datetime(df['order_date'], format='%d/%m/%Y'). Giving the format stops pandas guessing month-first and silently turning 03/04 into 4 March. After conversion, df['order_date'].dt.month and .dt.year work, and sorting is by date rather than by text.

Common mistakes and a practice habit

• Calling fillna and not assigning the result, then wondering why NaN is still there.
• Filling missing numbers with 0 when the value is unknown, which drags the mean down.
• Comparing with == None or == np.nan. NaN is not equal to anything, including itself. Use isna().
• Letting to_datetime guess the format on day-first data.

Open any CSV you have and run df.info() and df.isna().sum() before anything else. Fix one column at a time, and after each fix run df.dtypes and df.describe() to confirm the numbers look like numbers. Making this the first five minutes of every file is the habit employers notice.