PSYCH 647 Week 6 Evaluating an Appraisal Program Example

Did the New Appraisal System Work? A First-Year Evaluation at a Water Utility Using Reactions, Rating Quality and Results

[Student Name]

University of Phoenix

PSYCH/647: Human Performance, Assessment, and Feedback

Week 6 Assignment

[Instructor Name]

[Date]

The water utility, its evaluation data and findings are composites written for a model paper; research findings come from the sources listed.

A new system is a hypothesis: designers expect that its parts will change how people work. Evaluation tests that hypothesis. This final paper evaluates the first year of the performance management system designed in earlier weeks for field technicians at a water utility.

What Success Was Supposed to Mean

Before launch, the utility agreed on criteria for year one. Reactions: technicians would rate sessions, the system and its fairness more favorably than the old system. Rating quality: ratings would spread across the scale and vary across dimensions within technicians. Implementation: at least eighty-five percent of technicians would receive quarterly check-ins and an annual feedback conversation. Early outcomes: grievances related to appraisal would fall, and crew goals would show progress. Longer-term outcomes such as repeat leak calls and turnover would be tracked but not judged after one year.

Measuring Reactions

Keeping and Levy (2000) examined how to measure reactions to performance appraisal and tested a model including satisfaction with the appraisal session, satisfaction with the appraisal system, perceived utility, perceived accuracy and procedural and distributive justice. These reactions were related but distinct, and they formed a broader factor of appraisal reactions. The authors argued that reactions deserve attention because they influence acceptance and use of appraisal and may affect motivation and performance.

Following their model, our survey asked technicians to rate satisfaction with their feedback session, satisfaction with the system overall, usefulness, accuracy of their ratings and fairness of the process. We surveyed all technicians before launch about the old system and again after the first annual conversations.

Results: Reactions

Of 140 technicians, 112 responded. On a five-point scale, satisfaction with the session rose from 2.6 under the old system to 3.7, usefulness from 2.2 to 3.5, perceived accuracy from 2.8 to 3.4 and fairness from 2.7 to 3.6. Reactions were higher in districts where supervisors completed all quarterly check-ins.

Why Voice Matters

Cawley et al. (1998) meta-analyzed research on employee participation in performance appraisal and distinguished value-expressive participation, having a voice for its own sake, from instrumental participation, aimed at influencing the outcome. Participation was strongly related to satisfaction with the appraisal and perceived fairness, and value-expressive participation showed particularly strong relationships with satisfaction. The findings suggest that simply giving employees a voice, such as through self-assessments and goal setting, improves reactions.

Our survey comments echoed this: technicians frequently mentioned the self-assessment and crew goal setting as reasons the new system felt more fair.

The Supervisor Relationship

Pichler (2012) meta-analyzed research on the social context of performance appraisal and found that the quality of the relationship between rater and ratee was strongly related to employees' reactions, including satisfaction, perceived fairness, perceived accuracy and motivation to improve, often more strongly than features of the rating itself, such as whether ratings were favorable. Relationship quality also moderated how employees responded to less favorable ratings.

This finding fit our data: districts where supervisors held regular check-ins, which built relationships, showed better reactions even where ratings were lower.

Technicians in the districts with regular check-ins rated the system as fairer, even where their own ratings had gone down.

Results: Rating Quality

Under the old system, eighty-one percent of ratings were four. In year one, ratings spread across the scale: twelve percent were two, thirty-one percent three, forty-two percent four and fifteen percent five. Correlations between dimensions within technicians fell, suggesting less halo. Agreement between supervisors and lead technicians on the sample of thirty technicians was moderate, around .55, similar to research benchmarks, and lowest on adaptivity, suggesting that dimension needs clearer anchors.

Results: Implementation

Only sixty-one percent of technicians received all four quarterly check-ins, short of the eighty-five percent target. Supervisors cited workload during the summer main-break season. Annual conversations were completed for ninety-six percent.

Results: Early Outcomes

Appraisal-related grievances fell from seven in the year before to one. Eleven of fourteen crews met at least two of their three goals. Repeat leak calls and turnover will be examined after two more years.

What Supervisors Said

Supervisors' views matter as much as technicians', since they carry most of the workload. In focus groups, supervisors said the anchored scales made ratings easier to explain and that the separation of pay from feedback conversations removed the pressure they had felt to give everyone a four. Several said that giving their first honest two was uncomfortable but led to better conversations than expected. Their main complaint was time: quarterly check-ins with up to thirty technicians across shifts were hard to schedule, especially for supervisors covering night crews. Two supervisors admitted skipping check-ins with technicians they considered strong performers, which suggests that the system's informal parts are the first to slip under pressure.

Costs of the First Year

The first year cost about $38,000, mostly for rater training, supervisor time for check-ins and a part-time coordinator. Compared with the time previously spent on grievances, about six hundred staff hours in the prior year, the system appears to be close to paying for itself on that measure alone, though the comparison is rough.

Was the System Carried Out as Designed?

Program evaluation distinguishes a program that does not work from one that was never fully delivered. Check-ins, the heart of the everyday conversations the design emphasized, reached only sixty-one percent of technicians. The districts with full check-ins showed the strongest gains, which suggests that the design works when carried out but that implementation, not design, limited the overall results.

What We Cannot Conclude

There was no control group, and the utility hired new supervisors in two districts during the year, which could have affected reactions. Improvements in reactions might partly reflect relief at leaving an unpopular system. The evaluation shows promising change, not proof of effects on performance.

Comparing With Other Districts and Utilities

A stronger design for year two would compare the utility's results with a similar utility that has not changed its system, or stagger further changes across districts so that some serve as comparisons. The utility's water quality division, which kept the old system for one more year, offers a partial comparison for reactions, though its work differs from field operations.

Recommendations for Year Two

First, protect check-in time during the summer by scheduling them before and after peak season. Second, revise the adaptivity anchors and repeat frame-of-reference training. Third, coach supervisors in districts with low check-in rates. Fourth, keep the self-assessment and crew goals, which technicians valued most. Fifth, continue tracking long-term outcomes.

Reflection on the Course

This course followed one system from defining performance to evaluating results. The most important lesson was that the human side, raters' motives, feedback focus and supervisor relationships, mattered more than any form.

Conclusion

The first year produced better reactions, more informative ratings and fewer grievances, while check-in completion fell short. Research on appraisal reactions, participation and the supervisor relationship explains these results and points to protecting the everyday conversations on which the system depends.

References

Cawley, B. D., Keeping, L. M., & Levy, P. E. (1998). Participation in the performance appraisal process and employee reactions: A meta-analytic review of field investigations. Journal of Applied Psychology, 83(4), 615-633. https://doi.org/10.1037/0021-9010.83.4.615

Keeping, L. M., & Levy, P. E. (2000). Performance appraisal reactions: Measurement, modeling, and method bias. Journal of Applied Psychology, 85(5), 708-723. https://doi.org/10.1037/0021-9010.85.5.708

Pichler, S. (2012). The social context of performance appraisal and appraisal reactions: A meta-analysis. Human Resource Management, 51(5), 709-732. https://doi.org/10.1002/hrm.21499