RES 720 Week 3 Factorial ANOVA Example

Does Onboarding Help Every Shift Equally? A Factorial ANOVA With an Interaction

[Student Name]

University of Phoenix

RES/720: Statistical Research Methods and Design II

Week 3 Assignment

[Instructor Name]

[Date]

The learner, the company, its associates and all data are composites written for a model paper.

Week 2 found that rotating-shift associates at the composite Louisville logistics company pick fewer units per hour than fixed-shift associates. Meanwhile, the company began rolling out a structured onboarding program, which pairs each newcomer with a peer and lowers the pick target in week one, across its nine warehouses in three randomized steps. RQ2 asks whether shift and onboarding jointly influence 90-day productivity and whether the program's effect depends on shift. A factorial ANOVA tests both questions at once.

The Design

Shift has three levels and onboarding two, giving a 3 x 2 between-subjects design with six cells. Of about 900 associates hired during the rollout, 640 reached 90 days and have productivity data. Onboarding depended on whether the associate's warehouse had adopted the program at hiring, in a randomized order; shift was chosen by the associate.

Cell means, in units per hour:

Fixed days: current onboarding M = 117 (n = 110); new program M = 120 (n = 150).

Fixed nights: current M = 111 (n = 70); new M = 119 (n = 100).

Rotating: current M = 102 (n = 80); new M = 113 (n = 130).

The program gap widens from days to nights to rotating, a pattern that suggests an interaction.

Hypotheses

Main effect of shift: mean productivity is equal across shifts, averaging over onboarding. Main effect of onboarding: mean productivity is equal for the two programs, averaging over shift. Interaction: the onboarding effect is the same at every shift. Each null has an alternative stating that it does not hold.

Assumptions

Levene's test across the six cells was not significant, F(5, 634) = 1.8, p = .11, so variances are acceptably similar. A Q-Q plot showed residuals tracking the normal line closely. Cell sizes are unequal, so the learner uses Type III sums of squares, which test each effect adjusted for the others. Independence is the main concern, since associates are clustered in nine warehouses; a check appears below.

Results

Shift: F(2, 634) = 24.6, p < .001, partial eta squared = .072.

Onboarding: F(1, 634) = 38.2, p < .001, partial eta squared = .057.

Shift x onboarding: F(2, 634) = 3.9, p = .021, partial eta squared = .012.

All three effects are significant. The interaction is small by variance explained, but it changes how the program effect should be described.

A Note on Effect Size Labels

Levine and Hullett (2002) documented how researchers often reported partial eta squared, which SPSS printed by default, as if it were eta squared and explained that partial values can sum to more than 100 percent across effects and are not comparable across designs. The learner labels every value as partial and adds classic eta squared in an appendix table for readers who want the share of total variance.

Probing the Interaction

An interaction plot, with shift on the horizontal axis and separate lines for each onboarding condition, shows the lines diverging: almost touching on days and far apart on rotating shifts. Simple effects tests compare onboarding conditions within each shift, with a Bonferroni-adjusted alpha of .017:

Fixed days: difference 3 units per hour, p = .28, not significant.

Fixed nights: difference 8 units per hour, p = .006, d = 0.33.

Rotating: difference 11 units per hour, p < .001, d = 0.43.

The program helps most where support is thinnest. Night and rotating associates have fewer supervisors and trainers on site, so a peer buddy and a gentler start may fill a larger gap for them.

The program barely moves day-shift productivity, but it closes nearly half of the gap between rotating and day associates.

Why the Main Effects Need Care

The significant main effect of onboarding, averaged over shifts, suggests an overall gain of about 7 units per hour. Reported alone, it would overstate the benefit on days and understate it on rotating shifts. Because the interaction is significant, the learner describes the program's effect shift by shift.

Adding a Covariate

Prior warehouse experience, in months, is related to productivity and differs slightly across shifts. Including it in an ANCOVA reduced error variance; the interaction remained, F(2, 633) = 3.6, p = .028, and adjusted simple effects were nearly identical.

Miller and Chapman (2001) warned that ANCOVA is often misused to "control for" differences between groups that were not randomly formed and that removing a covariate's variance can remove part of the group difference itself, producing results that are hard to interpret. Here, onboarding was assigned by a randomized step order, so the covariate mainly sharpens precision; shift, however, was self-selected, so the learner does not claim that adjusting for experience makes shift groups equivalent.

Checking the Clustered Rollout

Hussey and Hughes (2007) described the design and analysis of stepped wedge trials, in which clusters cross from control to intervention in a randomized order, and showed that analyses should include time and cluster effects, since outcomes may drift over time and clusters differ. The learner reran the analysis as a mixed model with warehouse as a random effect and rollout period as a fixed effect. The program effect was slightly smaller, about 6 units per hour on average, and the interaction remained significant at p = .04. Its standard errors were larger, as expected with nine clusters, but the conclusions held.

A Survivor Check

The program also raised 90-day retention, from 63 to 71 percent. If the additional stayers were slower starters who would have quit under the old onboarding, they would pull the program group's mean down, making these estimates conservative. That possibility will be examined in Week 5's analysis of quitting.

What It Means for the Company

Leaders could prioritize the program for night and rotating shifts, where it has the largest effect, and consider a lighter version for days. The interaction also suggests a mechanism worth testing: support availability. If adding a night supervisor produced a similar gain, the program's buddy feature might be substituting for missing supervision. A small test at two warehouses, adding a night lead for one rollout step, could settle the question at modest cost.

Reporting in APA Style

A 3 x 2 ANOVA showed main effects of shift, F(2, 634) = 24.6, p < .001, partial eta squared = .07, and onboarding, F(1, 634) = 38.2, p < .001, partial eta squared = .06, qualified by a shift-by-onboarding interaction, F(2, 634) = 3.9, p = .021, partial eta squared = .01. Simple effects showed program gains on fixed nights and rotating shifts but not fixed days.

Conclusion

The factorial design answered a question one-way analyses could not: the onboarding program's benefit depends on shift, small on days and substantial on nights and rotating schedules. Correct effect size labels, an honest view of what a covariate can do and a check for the clustered rollout support that conclusion. Week 4 turns to multiple regression to weigh several predictors of productivity together.

References

Hussey, M. A., & Hughes, J. P. (2007). Design and analysis of stepped wedge cluster randomized trials. Contemporary Clinical Trials, 28(2), 182-191. https://doi.org/10.1016/j.cct.2006.05.007

Levine, T. R., & Hullett, C. R. (2002). Eta squared, partial eta squared, and misreporting of effect size in communication research. Human Communication Research, 28(4), 612-625. https://doi.org/10.1111/j.1468-2958.2002.tb00828.x

Miller, G. A., & Chapman, J. P. (2001). Misunderstanding analysis of covariance. Journal of Abnormal Psychology, 110(1), 40-48. https://doi.org/10.1037/0021-843X.110.1.40