RES 710 Week 4 Hypothesis Testing Example

Testing Claims About Member Experience: Hypothesis Testing, Errors and Power

[Student Name]

University of Phoenix

RES/710: Statistical Research Methods and Design I

Week 4 Assignment

[Instructor Name]

[Date]

The learner, the credit union, its members and all data are composites written for a model paper.

Weeks 2 and 3 described the composite Grand Rapids credit union's member survey and showed how much precision its 1,240 responses provide. The board now wants answers to two questions: has member advocacy declined since last year, and is satisfaction below the target set in the strategic plan? This paper uses those questions to work through the logic of hypothesis testing and the decisions that surround it.

The Logic of a Test

A hypothesis test begins by assuming that nothing has changed or that no difference exists, the null hypothesis, and asks how surprising the observed data would be under that assumption. If the data would be very unlikely, the researcher rejects the null in favor of the alternative. The significance level, alpha, is the risk of a false rejection the researcher is willing to accept, set before looking at the data, usually at .05.

Test 1: Has Advocacy Declined?

Last year's survey, run the same way, found that 45 percent of respondents were promoters, scoring 9 or 10 on likelihood to recommend. This year, 41 percent of 1,222 respondents were promoters.

H0: the population proportion of promoters is 0.45 (p = 0.45).

H1: the population proportion differs from 0.45 (p is not equal to 0.45).

A two-tailed test was chosen because the board wanted to know about change in either direction. The standard error under the null is sqrt(0.45 x 0.55 / 1,222) = 0.0142. The test statistic is z = (0.41 minus 0.45) / 0.0142 = -2.81, and the two-tailed p value is about .005. Because .005 is below .05, the null is rejected: the decline is unlikely to reflect sampling variation alone. The 95 percent confidence interval for this year's share, 38.2 to 43.8 percent, excludes 45 percent, telling the same story with more information.

Test 2: Is Satisfaction Below Target?

The strategic plan set a target mean satisfaction of 5.6 on the 7-point scale. The sample mean is 5.4 and the SD is 1.2.

H0: mean satisfaction is at least 5.6 (the population mean is 5.6 or higher).

H1: mean satisfaction is below 5.6 (the population mean is below 5.6).

A one-tailed test fits here because the board's question was directional before any data were collected. With a standard error of 0.034, t(1239) = (5.4 minus 5.6) / 0.034 = -5.88, and p < .001. Satisfaction is below target.

But Cohen's d, the difference divided by the standard deviation, is -0.2 / 1.2 = -0.17, a small effect. Members are, on average, a fifth of a scale point below target. With a large sample, even small differences become statistically significant, so the decision about whether this gap matters belongs to managers, informed by what a fifth of a point means for retention.

With 1,240 members, the test can see a difference of a fifth of a point; whether the board should care is a separate question.

What P Values Do and Do Not Mean

Wasserstein and Lazar (2016), writing for the American Statistical Association, explained that a p value measures how incompatible observed data are with one particular statistical model. It says nothing direct about the chance that the hypothesis itself is correct, nor about how large an effect is, and the statement warned against basing conclusions only on whether p falls under a cutoff such as .05. The learner's report therefore gives the estimate, interval and effect size for every test.

Cumming (2014) argued for shifting emphasis from significance tests to effect sizes, confidence intervals and meta-analytic thinking, since intervals convey both the size of an effect and its uncertainty. The board will see an interval chart of each year's promoter share, not just a verdict of significant or not.

Why the Same Survey Method Matters

Comparing this year with last year is fair only if both surveys used the same frame, invitation process and question wording. The learner confirmed that both years sampled transacting members in the same six-week season with identical items. Had the wording or timing changed, a difference in promoter share could reflect the method rather than members, and no test could tell the two apart.

False Alarms and Missed Declines

A Type I error here would be telling the board that advocacy fell when it did not, which might trigger a costly service overhaul. A Type II error would be missing a real decline, letting a problem grow. Lowering alpha to .01 would reduce false alarms but increase missed declines. The learner kept .05 for these questions because both errors carry similar costs, but she notes that a decision to spend heavily on a single result might justify a stricter standard.

Power for the Planned Comparisons

Power is the probability of detecting an effect of a given size if it exists. Cohen (1992) labeled a standardized mean difference of d = 0.2 small, d = 0.5 medium and d = 0.8 large and gave sample sizes needed for power of .80 at alpha .05. For a two-group comparison, detecting d = 0.3 requires about 175 members per group. Week 5 will compare app users, 248 respondents, with branch users, 645 respondents, so power exceeds .80 for a difference of that size. Detecting d = 0.2 would need about 393 per group, so a very small difference between app and branch members could be missed.

The Multiple Testing Problem

The regional managers asked for a test of each branch's satisfaction against the credit union's overall mean. Running 24 tests at alpha .05 would produce, on average, more than one false positive even if no branch truly differed. A Bonferroni correction divides alpha by the number of tests, giving .05 / 24 = .0021 per test. That approach is conservative and reduces power, so the learner will instead use a single model that compares branches together, reporting which branches stand out with adjusted intervals, and will treat branch rankings as signals for follow-up rather than final judgments.

Assumptions Behind the Tests

Both tests assume independent observations and, for the mean, a sampling distribution close to normal. The central limit theorem covers the second point given the sample size. Independence is less certain, since members of the same branch share conditions; Week 3's design effect means the true standard errors are somewhat larger than those shown, so the closing estimates will be computed with methods that adjust for clustering and weights.

Reporting the Results

In APA style, every test will appear with the test statistic, its df, the exact p value, an effect size and an interval, for example: mean satisfaction (M = 5.40, SD = 1.20) was below the target of 5.6, t(1239) = -5.88, p < .001, d = -0.17, 95% CI [5.33, 5.47]. Every reader then sees both how sure the result is and how large it is.

Conclusion

Hypothesis testing gave clear answers to the board's questions: advocacy has declined, and satisfaction sits below target, though by a small margin. Interpreting those results well required more than p values: confidence intervals, effect sizes, attention to both kinds of error, a power analysis for later comparisons and a plan for many tests. Week 5 will apply t tests to compare channels.

References

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. https://doi.org/10.1037/0033-2909.112.1.155

Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7-29. https://doi.org/10.1177/0956797613504966

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108