Hypothesis tests, p-values and confidence intervals

Inference questions check whether you can draw a careful conclusion from a sample. The calculations are short; the interpretation is where most marks are lost.

Hypothesis testing in four steps

1. State the null hypothesis, usually no effect or no difference, and the alternative.
2. Choose a significance level, commonly 0.05.
3. Calculate a test statistic and its p-value from the sample.
4. If p is below the significance level, reject the null. Otherwise, fail to reject it.

Say "fail to reject" rather than "accept". A non-significant result may simply mean the sample was too small to detect an effect.

What a p-value means

A p-value is the probability of getting results at least as extreme as yours if the null hypothesis were true. A p-value of 0.03 does not mean there is a 3% chance the null is true. Also, a very large sample can make a tiny, unimportant difference significant, so always look at the size of the effect as well.

The two possible mistakes are:

• Type I error: rejecting a true null (a false positive). Its probability equals the significance level.
• Type II error: failing to reject a false null (a false negative).

Confidence intervals

A 95% confidence interval for a mean is roughly the sample mean plus or minus 1.96 standard errors, where the standard error is s divided by the square root of n.

Example: 100 orders, mean 50 minutes, standard deviation 10. Standard error = 10 / 10 = 1. Interval = 50 plus or minus 1.96, so 48.04 to 51.96 minutes. Using the standard deviation instead of the standard error is the most common error in these questions.

Because of the square root, four times the sample size halves the standard error. Larger samples give narrower intervals; higher confidence levels give wider ones.

Correlation is not causation

A correlation coefficient r runs from -1 to 1 and measures the strength of a linear relationship. Two variables can move together because a third factor drives both, such as hot weather raising both ice cream sales and swimming accidents. Only a controlled experiment or careful design can support a causal claim.

Habit to build: after any result, write one sentence that states the finding and one that states what it does not prove.