Confidence intervals, p-values and hypothesis tests: one of each computed by hand, and what every sentence of the result does and does not mean

A 95 % confidence interval is a range computed from a sample that, by the method used, captures the true value in 95 % of samples; it is not a 95 % probability that the truth is in this one. A p-value is the probability of data at least this extreme if the null hypothesis were true; it is not the probability that the null is true. Those two sentences are where marks and meaning are lost. This page computes one interval, one t-test and one chi-square test by hand and says, for each, what the result does and does not mean.

The three objects, in one paragraph each

A confidence interval is an estimate with a margin: estimate ± margin of error. The estimate is what the sample says, the margin is how far the method allows the truth to be from it, and the confidence level (95 % here) is how often, over many samples, intervals built this way contain the true value.

Hypothesis testing starts from a null hypothesis H₀, a statement of no effect or no difference (the mean is 50; the die is fair), and an alternative H₁ that says otherwise. The data are summarised by a test statistic, a number that measures how far the sample sits from what H₀ predicts, often in units of its standard error.

A p value is the probability, if H₀ were true, of a test statistic at least as extreme as the one observed. A small p-value means the data would be surprising under H₀. At a significance level α (alpha), usually 0.05, a p-value below α leads to rejecting H₀; a p-value above it leads to failing to reject H₀, which is a weaker statement than accepting it, and the difference is one of the two sentences this page fixes.

In the first-year statistics course these are five ideas: idea 15, confidence intervals for a mean and a proportion; idea 16, hypotheses, test statistics and p-values; idea 17, significance level, Type I and Type II errors, and power; idea 18, the t distribution and one-sample t procedures; idea 21, chi-square tests for goodness of fit and independence. Before them is idea 14, sampling distributions, the standard error and the central limit theorem, which is where the standard errors below come from; after them is idea 19, comparing two means. In MDM4U the Grade 12 half of this is idea 24, estimating a proportion and the margin of error, which is Worked example 1; the MDM4U course map places it last in the course, and the first-year page says where a statistics course sits beside first-year calculus. AP Statistics asks all three computations in its inference units.

Worked example 1: a 95 % confidence interval for a proportion, line by line

Say 400 students are surveyed and 220 say they study with music on. The numbers are invented for the arithmetic; nothing about music is being claimed. How to calculate a confidence interval for the proportion of all students who do:

Line 1, the sample proportion: p̂ = 220/400 = 0.55.

Line 2, the standard error of a proportion, √(p̂(1 − p̂)/n): p̂(1 − p̂) = 0.55 × 0.45 = 0.2475; divided by n, 0.2475/400 = 0.00061875; square root, √0.00061875 ≈ 0.02487.

Line 3, the margin of error at 95 %, the z-score 1.96 times the standard error: 1.96 × 0.02487 ≈ 0.0488.

Line 4, the interval: 0.55 ± 0.049, that is (0.501, 0.599): between about 50 % and 60 %.

Line 5, the conditions, one sentence: the sample should be random, and the counts of successes and failures should both be large enough for the normal approximation, here 220 and 180, well above the ten of each that most introductory courses require (some texts use five).

What the sentence means. "We are 95 % confident the proportion is between 0.501 and 0.599" describes the method, not this interval. If you took many random samples of 400 and built an interval from each the same way, about 95 % of those intervals would contain the true proportion. This one either does or does not, and you cannot tell which.

The common wrong lines

"There is a 95 % probability the true proportion is between 0.501 and 0.599." The true proportion is a fixed number, not a random one. It is either in this interval or it is not, and the 95 % belongs to the procedure that produced the interval, not to the interval in front of you. Greenland and colleagues (2016) list this reading among the common misinterpretations of confidence intervals in their guide.

The standard error without the division by n. A student takes √(0.55 × 0.45) = 0.497 as the standard error and gets a margin of error near 1, an interval wider than the whole scale from 0 to 1. The standard error shrinks with the square root of the sample size; that is why a bigger sample gives a narrower interval. A margin of error larger than the estimate itself is a sign the n was dropped.

Worked example 2: a one-sample t-test, line by line

Suppose a school claims its students' mean study time is 50 minutes an evening, and a random sample of 25 students has mean 53 minutes and standard deviation 6 minutes. Is the claim consistent with the sample?

Line 1, the hypotheses: H₀: μ = 50. H₁: μ ≠ 50. The alternative is two-sided because the claim is "equals", and a mean below 50 would contradict it as much as one above.

Line 2, the standard error of the mean, s/√n: 6/√25 = 6/5 = 1.2.

Line 3, the test statistic: t = (53 − 50)/1.2 = 3/1.2 = 2.5, with n − 1 = 24 degrees of freedom.

Line 4, the two-sided p-value, the probability of |t| ≥ 2.5 with 24 degrees of freedom: about 0.020 (the distribution function gives 0.0197). On a t table, 2.5 lies between the two-sided critical values 2.492 for 0.02 and 2.797 for 0.01, so a table reader writes "between 0.01 and 0.02".

Line 5, the decision at α = 0.05: p < 0.05, so reject H₀. A sample mean at least this far from 50 would occur in about 2 % of samples if the true mean were 50; the data are unlikely under the claim.

Why t and not z. The population standard deviation σ is unknown and has been estimated by s from a small sample, so the normal distribution is replaced by the t distribution with n − 1 degrees of freedom, which is a little wider to allow for the extra uncertainty in s. When σ is known, the z procedure is the right one; when n is large (say over 30), t and z give nearly the same answer, and many courses use z then. That is the whole of when to use a t test vs z test.

The common wrong lines

"p = 0.02 means a 2 % chance the null hypothesis is true." The p-value is computed by assuming the null is true; it is the probability of the data given the null, not the probability of the null given the data. The second principle of the American Statistical Association's statement on p-values (Wasserstein and Lazar 2016) says exactly this: p-values do not measure the probability that the studied hypothesis is true, nor the probability that the data were produced by random chance alone.

A one-sided p-value for a two-sided claim. A student reads the one-tail area, 0.010, and reports it. The claim was "equals", so both tails count, and the p-value is 0.020. Halving the p-value without a one-sided alternative stated before the data were seen is a mark lost and a conclusion overstated.

"Accept H₀" when p is large. If the sample mean had been 51 and p had come out at 0.4, the right sentence is "fail to reject H₀": the data do not give evidence against the claim. A large p-value is an absence of evidence, not evidence that the mean is exactly 50, and "accept" claims the second.

Worked example 3: a chi-square goodness-of-fit test, line by line

A die is rolled 60 times. The six faces come up 8, 12, 9, 11, 14 and 6 times. Is the die fair? The chi square test for goodness of fit compares what was observed, O, with what a fair die would be expected to give, E.

Line 1, the expected count for each face under H₀ (the die is fair): 60/6 = 10 for each of the six faces.

Line 2, the contribution (O − E)²/E for each face: (8 − 10)²/10 = 4/10; (12 − 10)²/10 = 4/10; (9 − 10)²/10 = 1/10; (11 − 10)²/10 = 1/10; (14 − 10)²/10 = 16/10; (6 − 10)²/10 = 16/10.

Line 3, the test statistic, the sum: χ² = (4 + 4 + 1 + 1 + 16 + 16)/10 = 42/10 = 4.2, with 6 − 1 = 5 degrees of freedom.

Line 4, the comparison: the critical value at α = 0.05 with 5 degrees of freedom is 11.07, and 4.2 is far below it. The p-value is above 0.5 (the distribution function gives 0.52).

Line 5, the decision: fail to reject H₀. Counts this uneven, or more so, come up in about half of all sets of 60 rolls of a fair die. The data are consistent with a fair die. They do not prove the die is fair; a slightly loaded die would produce counts like these too.

The common wrong line

Forgetting to square the differences. The raw differences O − E are −2, 2, −1, 1, 4 and −4, which sum to zero, always, because the expected counts add to the same total as the observed ones. A student who skips the squaring gets χ² = 0 and concludes the fit is perfect. The square is what turns every miss into a positive contribution, as it does in a standard deviation. The other slip is using the observed counts as the expected ones, which also gives zero.

Significance level, errors and power, in one paragraph

The significance level α is the false-alarm rate you accept before seeing the data: at α = 0.05, a true null hypothesis is rejected in 5 % of samples, which is a Type I error. Failing to reject a null hypothesis that is false is a Type II error, and its rate is written β (beta). Power is 1 − β, the probability of detecting a real effect of a given size, and it grows with the sample size and with the size of the effect. A small sample that fails to reject has not shown there is no effect; it may simply have had too little power to see one. None of this needs a computation on a first-year test, but every sentence of a result rests on it.

Siblings to try cold

Two to do with the page closed; then check. Attempting first is chapter 1 of the free playbook, and testing yourself before you reread has the honest-marking rules.

1. A survey of 900 people finds 540 who say yes. Build the 95 % confidence interval for the proportion.
2. A claim says μ = 50. A random sample of 16 has mean 48 and standard deviation 4. Test the claim at α = 0.05.

Check

1. p̂ = 0.6; standard error √(0.6 × 0.4/900) ≈ 0.01633; margin of error 1.96 × 0.01633 ≈ 0.032; interval (0.568, 0.632).
2. Standard error 4/√16 = 1; t = (48 − 50)/1 = −2 with 15 degrees of freedom; two-sided p ≈ 0.064 (the table's critical value at 0.05 is 2.131, and |−2| is below it); at α = 0.05, fail to reject H₀. If you wrote "accept", reread the third wrong line of Worked example 2.

Where this sits in the course, and where it goes next

In the statistics course these are ideas 15 to 18 and 21 of 25, after sampling distributions (idea 14) and before the two-sample procedures (ideas 19 and 20). In MDM4U the margin of error is idea 24, the course's last idea.

The pieces come from earlier pages. The 1.96 is a z-score, the one with 2.5 % of the standard normal curve in each tail, from the normal distribution and z-scores. The s in the t-test is the sample standard deviation from the descriptive statistics page, divided by n − 1 for exactly the reason given there.

What comes next: comparing two means with independent or paired t procedures (idea 19), comparing two proportions (idea 20), the chi-square test of independence on a two-way table (the second half of idea 21), and regression (ideas 22 and 23). Each is one of this page's three computations with a different standard error. Sweller and Cooper (1985) found that studying worked examples paired with problems teaches a procedure faster than unguided attempts; work one of each kind as this page did, then a mixed set where you must decide which test the question wants.

Inference on one screen

• A confidence interval: p̂ ± 1.96 × √(p̂(1 − p̂)/n) at 95 %. "95 % confident" describes the method's hit rate over many samples, not this interval's chance of being right.
• A t-test: t = (x̄ − μ₀)/(s/√n), n − 1 degrees of freedom; two-sided unless the alternative was one-sided before the data.
• A chi-square test: χ² = Σ(O − E)²/E, categories − 1 degrees of freedom; the squares are what stop it summing to zero.
• The p-value: the probability of data at least this extreme if H₀ were true. Not the probability that H₀ is true.
• The decision: p < α, reject H₀; otherwise fail to reject. Never "accept".
• This page's numbers: interval (0.501, 0.599); t = 2.5, p ≈ 0.020, reject; χ² = 4.2, p ≈ 0.52, fail to reject.