HRM 326 Week 5 Evaluating Development Programs Example

Did the Supervisor Program Work? Four Levels of Evaluation, a Comparison Group of Outside Hires and a Return-on-Investment Estimate for Crossroads Fulfillment's First Cohort

[Student Name]

University of Phoenix

HRM/326: Employee Development

Week 5 Assignment

[Instructor Name]

[Date]

Crossroads Fulfillment and all results are composites written for a model paper; evaluation methods and research findings come from the sources listed and are stated generally.

A year ago, the composite Indianapolis logistics company launched its supervisor development program with 16 participants. Fourteen completed it. The chief operating officer asked one question at the quarterly review: was it worth the $104,000 it cost? Participants had rated the sessions highly, but she wanted to know whether graduates were better supervisors and whether the company was better off. Satisfaction surveys say whether people enjoyed a program; leaders need to know whether it changed what people do and what the business gets. This paper evaluates the program.

The Evaluation Framework

Kirkpatrick and Kirkpatrick (2016) set out a ladder of evidence that rises from how participants felt, to what they learned, to what they now do and finally to what the organization gains. Each level answers a different question, and higher levels matter more to the business but are harder to measure. Crossroads planned its evaluation before launch, collecting baseline data during the needs assessment so that changes could be measured.

Level One: Reaction

Participants rated each module on relevance, quality and how ready they felt to apply the skills. Average ratings were 4.5 out of 5 for labor planning and feedback modules and 3.8 for documentation, which participants found dry. Reactions help improve delivery but do not show whether the program worked.

Why Reactions Are Not Enough

Alliger et al. (1997) found in a meta-analysis that participants' reactions were only weakly related to learning and to later job behavior, though reactions about a program's usefulness related somewhat more strongly than reactions about enjoyment. A popular program can still fail to change behavior, so Crossroads measured further.

Level Two: Learning

Each participant was assessed against the learning objectives. On labor planning simulations, average accuracy improved from 18 percent off target before the program to 4 percent at the end, meeting the 5 percent standard. Scored role plays on corrective feedback rose from an average of 2.4 to 4.1 on a five-point scale. Documentation exercises improved from 60 percent correct to 92 percent.

Level Three: Behavior

Behavior was measured by mentor observations using checklists for start-up meetings and by managers' ratings of each graduate on the 24 supervisor tasks, compared with ratings before the program. Managers' ratings on people management tasks rose by about a full point on a five-point scale. Observers rated 12 of 14 graduates' start-up meetings as meeting the standard by the end of the rotation.

Level Four: Results

Ten graduates have been promoted to supervisor. The evaluation compares their shifts over their first six months with those of the ten supervisors hired from outside in the previous year during their first six months. Graduates' shifts had associate turnover of 14 percent annualized compared with 22 percent, productivity within 2 percent of standard compared with 6 percent below and recordable injuries at about half the rate. All ten graduates remain in their roles; four of the ten outside hires had left within six months.

Calculating Return on Investment

Phillips and Phillips (2016) describe return on investment as net program benefits divided by program costs. Costs were $104,000, including participants' paid time and shift coverage. Benefits include avoided supervisor turnover, about $9,000 per avoided departure, with four fewer departures expected than among outside hires, $36,000; reduced associate turnover on graduates' shifts, about 6 fewer departures at $4,000 each, $24,000; productivity gains worth about $70,000 in labor hours; and avoided injury costs of about $30,000. Total estimated first-year benefits are about $160,000. Net benefits are $56,000, a return of about 54 percent.

Testing the Assumptions

The estimate depends on attributing differences to the program. Graduates were selected for strong performance, so they might have outperformed outside hires anyway, a selection effect. They also knew the operation already. If only half the benefits are attributed to the program, the return falls to about negative 23 percent in the first year, though benefits from continued retention in later years would raise it. The evaluation therefore claims that the program likely pays for itself within two years rather than certainly within one.

What the Design Cannot Prove

Without random assignment, the comparison cannot rule out that graduates differ from outside hires in ways unrelated to the program. A stronger design for future cohorts would compare graduates with qualified applicants who were not selected because of limited spaces, though spaces should not be withheld only for research.

Negative Findings

The documentation module was rated poorly and two graduates still made documentation errors on their shifts. One participant left the program during the rotation, citing stress from supervising an unfamiliar crew.

Changes for the Next Cohort

The documentation module will use case competitions and short videos rather than lectures. The rotation will begin with two weeks of shadowing the outgoing supervisor. Selection will drop the software criterion, as found in Week 4.

Learning From the Dropout

The participant who left during the rotation agreed to an exit conversation. She said the program had prepared her for the tasks but not for the isolation of supervising a crew she did not know, on a shift where the outgoing supervisor had already left. Her account pointed to a design gap rather than a personal failing, and it led directly to the shadowing change for the next cohort. Treating a dropout as data rather than as a loss improved the program.

Costs Beyond the Budget

The evaluation also counted costs that do not appear in the program budget: mentors' time, about 200 hours across the year, and managers' time for monthly check-ins, about 100 hours. Including them raises total cost to about $118,000 and lowers the full-attribution return to about 36 percent. Leaders deserve the complete figure, even when it is less flattering.

Reporting to Leadership

The evaluation was presented to the operations committee with results at each level, the return estimate with its assumptions and the planned changes. The committee approved a second cohort of 18.

Conclusion

Crossroads' supervisor program produced strong reactions, measurable learning, observed behavior change and better shift results than outside hires, with an estimated first-year return of about 54 percent under full attribution. Honest testing of assumptions shows the program likely pays for itself within two years. Evaluation at every level, with negative findings reported, gives leaders confidence and guides the next cohort.

References

Alliger, G. M., Tannenbaum, S. I., Bennett, W., Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50(2), 341-358. https://doi.org/10.1111/j.1744-6570.1997.tb00911.x

Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick's four levels of training evaluation. ATD Press.

Phillips, J. J., & Phillips, P. P. (2016). Handbook of training evaluation and measurement methods (4th ed.). Routledge.