13.4.20 Type II errors and how they link to Type I · AI HL · 13.4: Hypothesis Testing

13.4.20

Type II errors and how they link to Type I

13.4.20

Type II errors and how they link to Type I

By the end of this section

Key points

The two errors

A Type I error is rejecting H0 when it is true. P(Type I error) is the actual significance level of the test.
A Type II error is not rejecting H0 when it is false. P(Type II error) = P(not in the critical region | the true distribution).
To find P(Type II error) you must be told the true value (the true mean, p or λ). Then use the true distribution, not the H0 distribution.
In context, say what each error means about the situation. Do not write only 'rejecting H0 when it is true'.

Finding P(Type II error)

Step 1: write the critical region under H0 (it is given, or find it from the significance level). Step 2: write the region where H0 is not rejected. Step 3: find its probability under the true distribution.
z-test of a mean: the true distribution of the sample mean is \(N(\mu_{true}, \sigma^2/n)\). Use σ/√n. For a two-tailed test the Type II region is between the two critical values.
Binomial and Poisson tests: for a critical region \(X \ge c\) the Type II region is \(X \le c - 1\). For \(X \le c\) it is \(X \ge c + 1\), which is 1 − P(X ≤ c).
A Poisson rate over a different length of time is scaled first: 0.4 per page over 10 pages gives mean 4.

Critical regions from a limit on an error

For a discrete test the actual significance level is the probability of the critical region under H0. It is usually less than the level you set.
Given a limit on P(Type I error), list cumulative probabilities under H0 and take the largest region that stays under the limit. A larger region means a smaller P(Type II error).
Given a limit on P(Type II error), use the true distribution to find the allowed boundary, then find P(Type I error) under H0 for that region.
When sampling without replacement from a small group, the counts are hypergeometric. Count the ways: C(red, k) × C(blue, n − k) ÷ C(total, n).

How the errors trade off

Lowering the significance level makes P(Type I) smaller and P(Type II) larger. Raising it does the opposite.
A larger sample reduces P(Type II) and leaves P(Type I) at the level. It is the only way to reduce P(Type II) without raising the level.
Which error matters more depends on the context. Missing a real illness is worse than a false alarm; convicting an innocent person is worse than letting a guilty one go free.

On the calculator

Normal cdf for a sample mean

Use the normal cdf with the true mean and standard deviation σ/√n of the sample mean. The lower and upper bounds are the ends of the Type II region.
For a region with no upper end use a very large bound such as 1E99; with no lower end use −1E99.

Screenshot and video to come

Binomial and Poisson cdf

The cdf gives P(X ≤ k) with the upper bound k. Enter n and p, or the Poisson mean.
For P(X ≥ k) use 1 − P(X ≤ k − 1).

Screenshot and video to come

A table of cumulative probabilities

To find a critical region from a limit, make a table of P(X ≤ k) for several k and read off the largest or smallest k that meets the limit.

Screenshot and video to come

Explore: Type I and Type II errors

A test of H₀: μ = 50 against H₁: μ > 50, with σ = 10. H₀ is rejected if the sample mean is above the critical value c. Drag the orange point to move c.

Sample size, n True mean, μ

Standard deviation of X̄, σ/√n
Critical value, c
P(Type I error)
P(Type II error)

Type I error: H₀ is rejected when it is true. Type II error: H₀ is not rejected when it is false. When n changes, c moves so that P(Type I error) stays the same.

Drag c along the axis.

Drag c to the right. What happens to the orange area? What happens to the teal area? Can you make both smaller by moving c?
Move the true mean closer to 50. What happens to P(Type II error)? Why is it hard to spot a small change in the mean?
Make n larger. What happens to the two curves, and to P(Type II error), when P(Type I error) is kept the same?
A medicine test: which error is worse, saying a medicine works when it does not, or missing one that does? What would you do to c?

Worked examples

Problems like the ones in class. Pick one, try it, then open the working.

Type II error: lower tail test of a mean

Paint tins are filled by a machine. The volume of paint is normally distributed with standard deviation 0.08 litres. A painter takes a sample of 12 tins to test, H0: μ = 5.00 against H1: μ < 5.00, and decides to reject H0 if the sample mean is below 4.96 litres. The true mean volume is 4.93 litres. Find the probability of a Type II error.

Working

The error
Type II error: do not reject H0 when it is false, so \(\bar X > 4.96\).

The test rejects below 4.96. A Type II error is everything on the other side of 4.96.
True distribution
\(\bar X \sim N(4.93, 0.08^2/12)\), standard deviation = 0.08/√12 = 0.0231

Use the true mean, 4.93, not the H0 mean. The sample mean has standard deviation σ/√n.
Probability
From the calculator: normal cdf, lower bound 4.96, upper bound 1E99, mean 4.93, standard deviation 0.0231: 0.0970

This is the blue area in the diagram: the true curve to the right of the critical value.
Answer
P(Type II error) = 0.0970

About 9.7% of the time this test would miss a mean as low as 4.93.

Type II error: two-tailed test of a mean

The height of a seedling after ten days is normally distributed with standard deviation 1.5 cm. A gardener tests H0: μ = 12.0 against H1: μ ≠ 12.0 at the 5% significance level, using a sample of 9 seedlings. The true mean height is 13.1 cm. Find the probability of a Type II error.

Working

Critical values
Under H0, \(\bar X \sim N(12.0, 1.5^2/9)\), standard deviation = 0.500. Each tail has 0.025. From the calculator: inverse normal, area 0.025, mean 12.0, standard deviation 0.500: 11.02; area 0.975: 12.98

Find the critical region first, using H0.
The error
Type II error: \(11.02 < \bar X < 12.98\)

H0 is not rejected between the two critical values.
True distribution
\(\bar X \sim N(13.1, 1.5^2/9)\)

Same standard deviation, new centre.
Probability
From the calculator: normal cdf, lower bound 11.02, upper bound 12.98, mean 13.1, standard deviation 0.500: 0.405

Most of the true curve lies above the upper critical value, so the test usually rejects.
Answer
P(Type II error) = 0.405

Keep the unrounded critical values in the calculator.

Type II error: a binomial test

A basketball player scores with 60% of her free throws. After changing her technique she takes 20 free throws to test whether her success rate has improved. She will conclude that it has improved if she scores at least 16. (a) State suitable hypotheses and describe a Type I and a Type II error in this context. (b) In fact her success rate is now 0.75. Find the probability of a Type II error.

Working

Hypotheses
H0: p = 0.6, H1: p > 0.6, where p is the probability that she scores with a free throw.

The test is for an improvement, so H1 is upper tailed.
The two errors
Type I: deciding her success rate has improved when it has not (p is still 0.6). Type II: deciding there is no evidence of improvement when it has improved.

Type I rejects a true H0. Type II accepts a false H0. Say what each means about the player.
Critical region
Reject H0 if X ≥ 16. A Type II error is X ≤ 15.

A discrete critical region: write the values on each side of the boundary.
True distribution
X ~ B(20, 0.75) From the calculator: binomial cdf, n = 20, p = 0.75, upper bound 15: 0.585

Use the true p, 0.75, not 0.6.
Answer
P(Type II error) = 0.585

For comparison, P(Type I error) = 0.0510 when p = 0.6.

Significance level and Type II error: a Poisson test

The number of typing errors on a page of a newsletter is modelled by a Poisson distribution with mean 0.4 per page. An editor thinks a new assistant makes more errors. She reads 10 of the assistant's pages and will conclude that the mean has increased if she finds at least 8 errors. (a) State the hypotheses and find the significance level of the test. (b) The assistant's true mean is 0.6 errors per page. Find the probability of a Type II error.

Working

Hypotheses and scale
H0: λ = 0.4 per page, H1: λ > 0.4 per page. For 10 pages, under H0: X ~ Po(4)

The test counts errors over 10 pages, so scale the rate: 10 × 0.4.
Significance level
P(Type I error) = P(X ≥ 8) = 1 − P(X ≤ 7) From the calculator: Poisson cdf, mean 4, upper bound 7: 0.94887, so 1 − 0.94887 = 0.0511

For a discrete test the significance level is the actual probability of the critical region under H0. It is not a round number.
True distribution
True mean 0.6 per page, so for 10 pages X ~ Po(6)

Scale the true rate in the same way.
Type II error
Type II error: X ≤ 7 From the calculator: Poisson cdf, mean 6, upper bound 7: 0.744

Not rejecting H0 means X is 7 or fewer.
Answer
Significance level = 0.0511 P(Type II error) = 0.744

A true mean 50% higher than 0.4 is missed most of the time with this sample.

Critical region from a Type I limit, then Type II

A bus company says that 70% of its buses arrive on time. A commuter group thinks the figure is lower and records 15 buses. They reject the company's claim if the number on time is in a critical region for which the probability of a Type I error is less than 0.08. (a) State the hypotheses and find the critical region that gives the smallest probability of a Type II error. (b) Given that the true proportion on time is 0.4, find that smallest probability of a Type II error.

Working

Hypotheses
H0: p = 0.7, H1: p < 0.7, where p is the proportion of buses on time. Under H0, X ~ B(15, 0.7); critical region X ≤ k

The commuters think the figure is lower: lower tail.
Try values of k
From the calculator: binomial cdf, n = 15, p = 0.7: P(X ≤ 6) = 0.0152; P(X ≤ 7) = 0.0500; P(X ≤ 8) = 0.131

Look for the largest k with probability below 0.08. A larger region means a smaller Type II error.
Critical region
The largest k with P(X ≤ k) < 0.08 is 7. Critical region: X ≤ 7

At k = 8 the Type I probability is 0.131, which is too big.
Type II error
True distribution X ~ B(15, 0.4). Type II error: X ≥ 8 P(X ≥ 8) = 1 − P(X ≤ 7) From the calculator: binomial cdf, n = 15, p = 0.4, upper bound 7: 0.787, so 1 − 0.787 = 0.213

Use the true p, 0.4.
Answer
Smallest P(Type II error) = 0.213

Any smaller region has a larger Type II probability.

Critical region from a Type II limit, then Type I

The number of dust particles in a cubic metre of air in a laboratory is Poisson with mean 2. After a ventilation fault a technician thinks the mean has increased. She counts the particles in one cubic metre and rejects H0 if the count is at least c. The true mean is now 7. She wants the probability of a Type II error to be less than 0.1. (a) State the hypotheses. (b) Find the critical region that gives the smallest probability of a Type I error, and state that probability.

Working

Hypotheses
H0: λ = 2, H1: λ > 2 (particles per cubic metre)

She thinks the mean has increased: upper tail.
Type II condition
Critical region X ≥ c. True distribution X ~ Po(7). Type II error: X ≤ c − 1, so we need P(X ≤ c − 1) < 0.1

Use the true mean for the Type II condition.
Try values
From the calculator: Poisson cdf, mean 7: P(X ≤ 2) = 0.0296; P(X ≤ 3) = 0.0818; P(X ≤ 4) = 0.173

A larger c makes Type II bigger and Type I smaller. Take the largest c that keeps Type II below 0.1.
Critical region
P(X ≤ 3) < 0.1 but P(X ≤ 4) is not, so c − 1 = 3. Critical region: X ≥ 4

Both conditions together fix the region.
Type I error
Under H0, X ~ Po(2): P(X ≥ 4) = 1 − P(X ≤ 3) From the calculator: Poisson cdf, mean 2, upper bound 3: 0.857, so 1 − 0.857 = 0.143

Now use the H0 mean.
Answer
Smallest P(Type I error) = 0.143

The trade-off in action: fixing Type II fixed the region, and the region fixes Type I.

Errors when sampling without replacement

A bag contains 12 counters, r of them red and the rest blue. Maya knows that r is either 3 or 8 but not which. She takes 4 counters from the bag without replacement and tests H0: r = 3 against H1: r = 8. She rejects H0 if at least 3 of the counters are red. (a) Find the probability that she makes a Type I error. (b) Find the probability that she makes a Type II error.

Working

Type I error
Reject H0 when H0 is true (r = 3): P(X ≥ 3) = P(X = 3), since only 3 red counters exist. P(X = 3) = C(3,3) × C(9,1) ÷ C(12,4) = 9/495

Choose the red counters from the red ones and the blue from the blue ones. C(12,4) is all the ways to take 4.
Value
P(Type I error) = 0.0182

Sampling without replacement: use counting, not B(4, 3/12).
Type II error
Do not reject H0 when r = 8: P(X ≤ 2) = 1 − P(X ≥ 3) P(X = 3) = C(8,3) × C(4,1) ÷ C(12,4) = 224/495 P(X = 4) = C(8,4) ÷ C(12,4) = 70/495

Under H1 the bag has 8 red and 4 blue. Easier to find the opposite.
Answer
P(Type II error) = 1 − (224 + 70)/495 = 0.406

A simple H1 (r = 8) gives a definite Type II probability.

How sample size and level change Type II

A test of H0: μ = 50 against H1: μ > 50 is carried out on a normal population with standard deviation 6, at the 5% significance level. The true mean is 53. Find the probability of a Type II error when the sample size is n = 36, and compare it with n = 9 and with a 1% level.

Working

n = 9
Standard deviation 2.00; critical value (inverse normal, area 0.95, mean 50, unrounded): 53.29; P(Type II) = P(X̄ < 53.29 | μ = 53) = 0.558

The starting point.
n = 36
Standard deviation 6/√36 = 1.00; critical value: 51.645; P(Type II) = P(X̄ < 51.645 | μ = 53) = 0.0877

A larger sample makes the sample mean less spread out. The two curves overlap less.
Same level
P(Type I error) = 0.05 in both tests

The level was fixed, so only n changed. Increasing n lowers P(Type II) without raising P(Type I).
A 1% level
n = 9, 1% level: critical value 54.653; P(Type II) = P(X̄ < 54.653 | μ = 53) = 0.796

Lowering the level to 1% raised P(Type II) from 0.558 to 0.796. Lower α means higher P(Type II).

Menu

Hypothesis Testing

Type II errors and how they link to Type I

By the end of this section

Key points

The two errors

Finding P(Type II error)

Critical regions from a limit on an error

How the errors trade off

On the calculator

Normal cdf for a sample mean

Binomial and Poisson cdf

A table of cumulative probabilities

Explore: Type I and Type II errors

Worked examples

Type II error: lower tail test of a mean

Working

Type II error: two-tailed test of a mean

Working

Type II error: a binomial test

Working

Significance level and Type II error: a Poisson test

Working

Critical region from a Type I limit, then Type II

Working

Critical region from a Type II limit, then Type I

Working

Errors when sampling without replacement

Working

How sample size and level change Type II

Working

Choose a skill

Quiz