PSYCH 664 Week 4 Quantitative Analysis Example

Analyzing the Pilot: Group Comparisons, Effect Sizes, Confidence Intervals and What a p Value Does Not Say

[Student Name]

University of Phoenix

PSYCH/664: Research Methods and Statistics in Psychology

Week 4 Assignment

[Instructor Name]

[Date]

The senior community, the pilot data and all figures are composites written for a model paper; statistical guidance comes from the sources listed.

Statistics turn data into evidence, but only when analyses fit the design and results are interpreted with care. This paper analyzes data from a twelve-week pilot of the phone-call program designed in earlier weeks.

The Pilot

Before the full study, our community ran a pilot to test procedures and estimate effects. Forty-two residents aged seventy and older who scored at least moderate on the three-item loneliness screen were randomly assigned to a twenty-minute call each week, for twelve weeks, from a college student volunteer who had been trained or to a waitlist. Loneliness was measured with the twenty-item UCLA scale, which ranges from 20 to 80, at baseline and twelve weeks by a staff member who did not know group assignments.

Describing the Data

Of forty-two residents, thirty-eight completed the twelve-week measure, twenty of twenty-one in the call group and eighteen of twenty-one on the waitlist. Two waitlist residents moved to memory care and one declined to continue; one call-group resident was hospitalized. At baseline, mean loneliness was 51.2 (standard deviation 8.4) in the call group and 50.6 (standard deviation 9.1) in the waitlist group, similar as expected after randomization. At twelve weeks, means were 45.1 (standard deviation 8.9) and 49.8 (standard deviation 9.4).

Checking Assumptions

Comparing groups with analysis of covariance assumes that residuals are roughly normally distributed, that variances are similar across groups and that the relationship between baseline and follow-up scores is similar in both groups. Histograms and a normality test suggested approximately normal residuals; Levene's test did not indicate unequal variances; and the interaction between group and baseline loneliness was small and not significant, supporting the assumption of parallel slopes.

Choosing the Analysis

Comparing only twelve-week scores would ignore baseline differences and waste information. Comparing change scores is reasonable, but analysis of covariance, with twelve-week loneliness as the outcome and baseline loneliness as the covariate, generally offers more power and adjusts for chance baseline differences in randomized trials. Group is the factor of interest.

Results

Baseline loneliness strongly predicted twelve-week loneliness. Adjusting for baseline, the call group's mean loneliness at twelve weeks was 4.9 points lower than the waitlist group's, (95 percent confidence interval: 0.6 to 9.2 points lower), F(1, 35) = 5.34, p = .027. Dividing the adjusted difference by the pooled standard deviation gives a standardized effect of about 0.56, with a confidence interval roughly from 0.07 to 1.05.

The calls appear to help, but the confidence interval runs from a trivial effect to a large one.

Interpreting the Effect Size

Cohen (1992) offered a primer on statistical power, presenting conventions for small, medium and large effect sizes across common tests, with standardized mean differences of two tenths, one half and eight tenths of a standard deviation marking the three levels, and tables for sample sizes needed to detect them. Cohen cautioned that these conventions were offered as rough guides when better information was unavailable, not as fixed standards.

Our estimate of 0.56 is near Cohen's medium benchmark, but practical meaning matters more. A five-point drop on the UCLA scale represents a noticeable shift, roughly from frequently to sometimes feeling isolated on several items.

What the p Value Does Not Say

Wasserstein and Lazar (2016) summarized the position the American Statistical Association took on p values. Its six principles, in my words, say that a p value tells how poorly data fit a stated model, not the chance that a hypothesis is right or that luck alone produced the data; that a decision should never hinge on which side of a cutoff p falls; that honest inference needs every analysis reported; and that a small p says nothing about whether an effect is big or matters.

A p of .027 does not mean there is a 97 percent chance that calls reduce loneliness, and had p been .06, it would not mean calls do nothing.

Estimation Over Dichotomy

Cumming (2014) argued for the new statistics, emphasizing estimation, effect sizes, confidence intervals and meta-analysis, over null hypothesis significance testing. Cumming recommended reporting effect sizes with confidence intervals, interpreting the full range of plausible values, planning studies for precision and accumulating evidence across studies. Following this approach, I report that the best estimate of the calls' effect is a moderate reduction in loneliness, with plausible values ranging from very small to large.

Missing Data

Four of forty-two residents, about ten percent, lacked twelve-week data. The main analysis used complete cases. As a sensitivity analysis, I carried forward baseline scores for missing residents, a conservative assumption of no change; the adjusted difference was 4.2 points, still favoring calls. Because two waitlist dropouts moved to memory care, missingness may be related to decline, which I note as a limitation.

Why Not Compare Change Scores?

A common alternative is to subtract each resident's baseline from the twelve-week score and compare average change between groups with a t test. Change scores are easy to explain, and in a randomized trial they give an unbiased estimate of the effect. However, because baseline and follow-up scores are correlated but not perfectly, analysis of covariance usually estimates the same effect with a smaller standard error, which means more precision from the same residents. In our data, the change-score comparison gave a similar difference with a slightly wider confidence interval, which supports reporting the covariance analysis as the main result and the change scores as a check.

Explaining the Result to Residents and Staff

Statistics have to make sense to the people who will use them. When I presented the pilot to residents and staff, I avoided the language of significance and said instead that residents who received calls became, on average, noticeably less lonely than those waiting, that the size of the improvement was uncertain because the group was small and that the full study would give a clearer answer. Several residents said they had suspected as much from their own experience, which is also a reason to test the program properly rather than rely on impressions.

Secondary Outcomes

Depressive symptoms showed a smaller difference favoring calls, with a confidence interval including zero. Activity attendance did not differ. With a small sample and several outcomes, these results should be viewed as exploratory.

What the Pilot Changed for the Full Study

The pilot also tested procedures. Callers completed ninety-four percent of scheduled calls, residents rated the calls highly and the blind assessor reported only one resident who mentioned the calls during assessment. The main procedural change is to recruit more men, who made up only six of the forty-two participants, by inviting them through the community's woodworking and veterans groups.

Conclusion

The pilot suggests that weekly student phone calls reduced loneliness by a moderate amount, but the estimate is imprecise. Reporting effect sizes with confidence intervals and interpreting p values carefully supports the plan to run the full study with about ninety residents, which will narrow the interval and clarify the size of the effect.

References

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159. https://doi.org/10.1037/0033-2909.112.1.155

Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7-29. https://doi.org/10.1177/0956797613504966

Wasserstein, R. L., & Lazar, N. A. (2016). The ASA's statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108