Transforming columns: apply, map, str methods and avoiding slow loops

Most analysis tasks are "make a new column from an old one": a full name from first and last, a category from a price, a clean lower-case email. pandas has three ways to do this, and screening questions test whether you know which is appropriate. The order of preference is: vectorised operation, then a string accessor or map, then apply, and almost never a Python for loop over rows.

Vectorised operations first

Arithmetic and comparisons work on whole columns at once. Example 1: df['total'] = df['qty'] * df['price'] computes every row in one step. df['big'] = df['total'] > 1000 gives a True/False column. For conditional values, np.where(df['total'] > 1000, 'large', 'small') replaces an IF, and np.select([cond1, cond2], ['A', 'B'], default='C') handles several bands.

These run in compiled code and are many times faster than looping. If a solution contains for i in range(len(df)), there is almost always a vectorised version.

String methods with .str

Text columns get the .str accessor. Example 2: df['email'] = df['email'].str.strip().str.lower() cleans spacing and case. df['name'].str.split(' ', n=1, expand=True) splits into two columns. df['city'].str.contains('Chennai', case=False, na=False) gives a filter; the na=False stops NaN rows raising an error when the result is used as a boolean index. .str.len(), .str.replace() and .str.startswith() cover most other needs.

map and apply

Series.map takes a dictionary or a function and applies it to each value. Example 3: df['region'] = df['state'].map({'TN': 'South', 'KA': 'South', 'DL': 'North'}). States not in the dictionary become NaN, which is a useful signal that your mapping is incomplete.

Series.apply(func) runs a function on each value. DataFrame.apply(func, axis=1) runs it on each row, which is slower but lets the function see several columns. Example 4:

def band(row):
    if row['age'] < 18:
        return 'minor'
    return 'adult' if row['income'] > 0 else 'dependent'

df['band'] = df.apply(band, axis=1)

Use apply when the logic is too awkward for np.where. For simple cases, np.where is both shorter and faster.

Common mistakes and a practice habit

• Looping over rows with iterrows for simple arithmetic.
• Forgetting .str and calling .lower() on the Series directly, which raises an error.
• Using map with a dictionary and not checking for the NaN rows it creates.
• Assigning to a filtered slice such as df[df['x'] > 0]['y'] = 1, which triggers a warning and may change nothing. Use df.loc[df['x'] > 0, 'y'] = 1.

Take a column of messy text and a column of numbers and write five transformations: one arithmetic, one np.where, two .str chains and one map with a dictionary. Time each against a for loop using %%timeit in a notebook. The difference teaches the preference order better than any rule.