Missing data · Pandas

Real data has gaps: a skipped question, a broken sensor, a field nobody filled in. pandas marks them as missing, usually shown as NaN ("not a number").

Python

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Leo", "Mei", "Raj", "Zoe"],
    "age": [21, np.nan, 24, 35, np.nan],
    "city": ["Mumbai", "London", None, "Delhi", "Rome"],
    "score": [92, 78, np.nan, 88, 95],
})
print(df)
print(df.isna().sum())        # missing values per column
print(df.isna().any(axis=1))  # which rows have at least one gap

isna() gives True wherever a value is missing; sum() then counts them per column. This is one of the first things to run on any new dataset. (A missing text value may display as None or NaN, depending on the pandas version.)

Dropping missing values

Python

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Leo", "Mei", "Raj", "Zoe"],
    "age": [21, np.nan, 24, 35, np.nan],
    "city": ["Mumbai", "London", None, "Delhi", "Rome"],
    "score": [92, 78, np.nan, 88, 95],
})
print(df.dropna())                    # only rows with no gaps at all
print(df.dropna(subset=["score"]))    # only rows missing a score are dropped
print(df.dropna(thresh=3))            # keep rows with at least 3 values
print(df.dropna(axis=1))              # drop columns that have any gap

Dropping is simple, but it throws information away: dropna() kept only two of five rows here. Use subset to drop only for the columns that matter to your question.

Filling missing values

Python

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "name": ["Asha", "Leo", "Mei", "Raj", "Zoe"],
    "age": [21, np.nan, 24, 35, np.nan],
    "city": ["Mumbai", "London", None, "Delhi", "Rome"],
    "score": [92, 78, np.nan, 88, 95],
})
filled = df.fillna({"age": df["age"].mean(), "city": "Unknown", "score": 0})
print(filled)

fillna with a dictionary sets a different fill value for each column. Filling with the column's mean or median keeps averages roughly unchanged; filling with 0 only makes sense when missing really means zero.

Filling from neighbours

Python

import numpy as np
import pandas as pd

temps = pd.Series([21.0, np.nan, np.nan, 24.5, np.nan, 23.0],
                  index=pd.date_range("2024-07-01", periods=6))
print(pd.DataFrame({
    "raw": temps,
    "ffill": temps.ffill(),             # carry the last known value forward
    "interpolate": temps.interpolate(), # draw a straight line between known values
}))

For data in time order, like readings or prices, the neighbouring values are usually the best guess. ffill repeats the last known value; interpolate fills the gaps in a straight line.

How missing values behave

Python

import numpy as np
import pandas as pd

s = pd.Series([10, np.nan, 30])
print(s.sum(), s.mean(), s.count())     # the NaN is skipped
print(s.dtype)                          # float64, even though the values are whole numbers

counts = pd.Series([10, None, 30], dtype="Int64")
print(counts)                           # whole numbers that allow gaps, shown as <NA>

Most calculations skip missing values: the mean here is (10 + 30) / 2 = 20, not 40 / 3.
NaN is a float, so one gap turns a column of integers into floats. The Int64 type, with a capital I, keeps whole numbers and allows missing values.