HRM 552 Week 6 Evaluating Organizational Development Example

Did Cycle Time Return to 14 Days? Evaluating the Training and Organizational Development Program at Bayou Coastal Insurance

[Student Name]

University of Phoenix

HRM/552: Organizational Training and Development

Week 6 Assignment

[Instructor Name]

[Date]

Bayou Coastal Insurance and all evaluation data are composites written for a model paper.

For nine months, the composite insurer Bayou Coastal has worked through a program to reduce homeowners claim cycle time from 23 days back to 14. It restored payment authority, added temporary adjusters, formed a catastrophe team, tied bonuses to speed and accuracy, ran six weeks of blended training and worked on employee relations problems. Leadership now asks what worked. An evaluation that cannot separate the effect of training from everything else happening at the same time will either overstate or understate what the training achieved. The evaluation below answers that question.

The Evaluation Framework

The evaluation uses Kirkpatrick and Kirkpatrick's (2016) four levels, which climb from participant satisfaction through knowledge gained and changed work habits up to business outcomes. Phillips and Phillips (2016) add a fifth level, return on investment, which converts results to money and compares them with costs. The evaluation plan was set in Week 3, before training began, so baseline data were collected.

The Evaluation Design

Because training rolled out in three waves of 80 adjusters, about six weeks apart, the third wave served as a comparison group while the first two were trained. Shadish et al. (2002) describe such staggered designs as a practical way to approximate experimental comparison when random assignment is not possible. Trends before and after each change were also examined.

Level One: Reaction

Surveys after each session showed that 87 percent of adjusters rated the training useful, compared with 41 percent for the original platform training. The sandbox and peer coaches received the highest ratings. Veterans rated the modules on complex claims highly. Reactions do not prove learning but suggest engagement.

Level Two: Learning

Sandbox assessments measured whether adjusters could perform each feature correctly. Scores rose from an average of 58 percent on the pretest to 91 percent after training. Arthur et al. (2003) found that training effects on learning measures tend to be larger than effects on behavior and results, so a program that looks strong on tests may still show little change at work.

Level Three: Behavior

Platform data showed that use of automated document requests on eligible claims rose from 60 percent to 88 percent, diary reminders from 45 to 84 percent and correct reassignments within 24 hours from 70 to 93 percent. Adjusters in the first two waves increased feature use sharply after training, while the third wave changed little until its own training, which supports training as the cause.

Level Four: Results

Average homeowners cycle time fell from 23 days to 15.2 days over nine months. Complaints fell from 3.4 to 2.2 per 1,000 claims, and audit error rates from 6 to 3.8 percent. Adjuster turnover fell from 28 to 21 percent on an annualized basis.

Isolating the Effects of Each Change

The changes took effect at different times, allowing estimates. Approval queue time fell from 3.5 days to 0.6 days within weeks of restored authority, accounting for about 2.9 days. Caseload reductions from temporary adjusters coincided with a drop of about 2 days. The comparison of trained and untrained waves showed a difference of about 1.3 days in cycle time. The bonus change, introduced last, coincided with a further 0.6-day drop, though only one quarter of data is available.

Behavior Beyond the Platform

Not every behavior shows up in system data. Supervisors were asked to rate coaching conversations, and adjusters were surveyed on whether supervisors reviewed aging dashboards with them weekly. Seventy-two percent said their supervisor did so most weeks, up from almost none before the program, though the rate varied from 50 to 90 percent across offices. Offices where supervisors coached regularly showed larger gains in feature use, consistent with research on supervisor support and transfer from Week 3.

Sustaining the Gains

A common pattern after training is decline as attention moves elsewhere. Three months after the first wave finished, its feature use remained at 87 percent, suggesting the dashboards, coaching and bonus are sustaining behavior. The evaluation will check again at 12 months, and supervisors' scorecards will continue to include team feature use and cycle time so the behaviors remain visible.

Program Costs

Costs over nine months were about $310,000 for training and development, $1.1 million for temporary adjusters, $85,000 for tablets and $60,000 for consulting on the bonus redesign, a total of about $1.56 million.

Program Benefits

Week 1 estimated the full 9-day gap cost about $4.8 million a year, or roughly $530,000 per day of cycle time. The 7.8-day improvement is worth about $4.1 million a year. Training's 1.3-day share is worth about $690,000.

Return on Investment

For the whole program, return on investment is ($4.1 million minus $1.56 million) divided by $1.56 million, about 163 percent in the first year. For training alone, it is ($690,000 minus $310,000) divided by $310,000, about 123 percent. These figures depend on the cost-per-day estimate and should be treated as approximate.

What Did Not Work

Some elements fell short. The development track for future team leads lost 4 of 15 participants to workload pressures. Engagement recovered only to 55 percent favorable, below the 60 percent target. Cycle time remains above the 14-day goal. Desk adjusters' turnover did not improve.

Explaining the Shortfalls

Interviews suggest that development participants could not balance stretch assignments with caseloads, and desk adjusters felt the changes focused on field adjusters. Trust takes longer to rebuild than processes. The remaining cycle time gap appears related to delays from outside contractors' estimates.

Limits of the Evidence

The comparison waves were not randomly assigned, and other factors such as a milder storm season may have helped. The bonus effect rests on limited data. Monetary estimates depend on assumptions about cost per day. The conclusions are strong for authority and training and more tentative for the bonus.

Recommendations

Keep restored authority, audit sampling and the catastrophe team. Make temporary adjuster capacity permanent at a smaller scale. Continue the bonus and review after two more quarters. Protect development time for team lead candidates. Extend attention to desk adjusters, including a review of their workloads. Work with contractors on estimate turnaround.

Reporting Back to Employees

Findings will go to adjusters and supervisors at office meetings, including the shortfalls. Employees who took part deserve to know what their effort achieved, and showing that leadership reports disappointing results honestly supports the trust rebuilding described in Week 5. The design team from Week 4 will review the findings and propose adjustments before the next cycle begins.

Continuing the Cycle

Following the action research model from Week 2, these results begin the next cycle of diagnosis. The division will set a new target, investigate remaining delays and repeat the evaluation in six months.

Conclusion

The program reduced cycle time from 23 to 15.2 days, with restored authority and lower caseloads contributing most and training about 1.3 days. Return on investment is strong. Reporting shortfalls in engagement, development and desk adjuster retention honestly points to the next round of work. Evaluation, done this way, becomes part of how the organization continues to develop.

References

Arthur, W., Jr., Bennett, W., Jr., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88(2), 234-245. https://doi.org/10.1037/0021-9010.88.2.234

Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick's four levels of training evaluation. ATD Press.

Phillips, P. P., & Phillips, J. J. (2016). Handbook of training evaluation and measurement methods (4th ed.). Routledge.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.