ORG 727 Week 8 Evaluating Intervention Results Example

Did the Intervention Work? Evaluating a Credit Union's Member Conversion Change Six Months After Rollout

[Student Name]

University of Phoenix

ORG/727: Organizational Diagnosis and Intervention

Week 8 Assignment

[Instructor Name]

[Date]

Saguaro Federal Credit Union, its people, results and figures are composites written for a model paper.

Over seven weeks, this course followed a diagnosis and intervention at Saguaro Federal Credit Union, the invented Tucson institution. The diagnosis traced stalled membership growth to lost loan applicants; the intervention rebuilt how approved loan applicants become members, with conversion specialists, shared incentives and a digital link. The rollout finished six months ago. This paper evaluates the results and recommends what the credit union should do next.

Evaluation Questions

Did conversion of approved non-member applicants rise, and by how much?

Did the intervention, rather than other events, cause the change?

What happened to members' experience, staff and other units?

Should the credit union sustain, adjust, extend or end the intervention?

Evaluation Design

Shadish et al. (2002) describe quasi-experimental designs that strengthen causal inference without random assignment by adding comparison groups, multiple pretests and staggered introduction of the treatment. The evaluation uses three such features. First, an interrupted time series: monthly conversion for 24 months before and 12 months after the prototype. Second, staggered rollout: Tucson applicants began two months before the rest of Arizona, so if conversion rose in Tucson first and elsewhere only when the change arrived, the intervention is the likely cause. Third, a comparison: a credit union of similar size in New Mexico shared anonymized conversion data through a trade association.

Primary Results

Conversion rose from an average of 41 percent before the change to 66 percent across the six months after full rollout. In Tucson, conversion rose in the month the specialists started; elsewhere in Arizona, it stayed flat until the change arrived two months later, then rose. The comparison credit union's conversion moved from 44 to 46 percent over the same period. On an annual basis, the gain is about 1,450 additional members, short of the target of 70 percent conversion but well above the starting point.

Conversion rose region by region exactly as the rollout arrived, which is hard to explain any other way.

Measuring Implementation as Well as Outcomes

Outcomes alone cannot show why an intervention worked or failed. The evaluation also checked whether the intervention was carried out as designed. Specialists called 91 percent of approved applicants within two hours, near the 95 percent target. Video completion worked for 88 percent of those who chose it, with the phone alternative covering most of the rest. Lending and branch staff used the dashboard daily in 30 of 38 branches; the 8 that did not had lower conversion in their areas, about 58 percent against 67 percent elsewhere. That difference suggests that the shared information, not only the specialists, contributed to results.

Alternative Explanations

Two events could explain part of the gain. A large competitor raised auto loan rates by half a point in month two, which might have made applicants more willing to complete their credit union loans. Regional auto sales rose about 5 percent. The staggered pattern argues against both, since both affected all regions at once, and the comparison credit union, facing the same market, saw little change. The intervention most likely caused most of the gain; the rate change may account for a few points.

Outcomes at Several Levels

Members: satisfaction among converted applicants averaged 4.5 of 5; time to funding averaged 0.9 days. Twelve-month retention of converted members is not yet known.

Staff: specialists reported high engagement, 4.4 of 5, and none left. Lending officers' views moved from negative before the incentive change to neutral, 3.1, afterward. Branch staff scores on cross-unit cooperation rose from 2.6 to 3.3.

Organization: new members per year are on pace for about 27,000, up from 25,400, ending the three-year decline. Net income from new member relationships is projected at about $3.4 million a year against costs of $2.4 million in the first year and about $1.2 million in later years.

Types of Change

Golembiewski et al. (1976) distinguished alpha change, real change on a stable dimension; beta change, recalibration of the scale people use; and gamma change, a redefinition of the concept being measured. Most results here are alpha changes. But interviews suggest gamma change among branch staff: before, they saw their role as opening accounts for people who walked in; now several describe it as helping members wherever they start. Their survey scores on role clarity barely moved, which may reflect that they now judge clarity against a different idea of the role.

Comparing Results With the Diagnosis

The evaluation also tests the diagnosis. Week 4 ranked disconnected processes and missing ownership as the main causes. If that ranking was right, fixing them should close much of the gap, and it did: conversion rose by 25 points. The remaining gap to the 70 percent target appears among applicants who could not be reached by phone, suggesting a cause the diagnosis did not rank highly, applicants' preference for text messages. That finding feeds the recommendation to test a text option.

Unintended Effects

Branch account openings fell a further 9 percent as conversions moved to specialists, intensifying branch managers' concerns about relevance. Contact center volume fell 12 percent, a positive side effect. Fraud review of the first 500 video-verified accounts found two suspicious cases, a rate comparable to branches.

Confidence in Conclusions

Confidence is high that conversion rose and that the intervention caused most of the rise. It is moderate on financial projections, which depend on member retention. It is low on the gamma change interpretation, which rests on a few interviews.

Costs Against Benefits

First-year costs came in at $2.3 million, slightly under budget. If converted members stay at the credit union's historical rate, the intervention pays back in about eighteen months. The estimate is sensitive to retention: if converted members leave twice as fast as other new members, payback stretches to nearly three years. Tracking their retention over the next year is therefore the most important open question.

Recommendations

Sustain the conversion process and incentives. Adjust by adding two specialists to reach the 70 percent target and by testing a text-message option for applicants who miss the call. Extend the approach to mortgage applicants, where a similar gap exists. Address branches' role through a separate diagnosis of branch purpose rather than reversing the intervention.

Reflection

Armenakis and Bedeian (1999) reviewed a decade of change research and called for attention to content, context, process and outcomes together. This engagement taught me that diagnosis earns its value by testing leaders' explanations before money is spent. All three executives' fixes would have cost more and missed the cause. I would gather member records earlier next time, since they proved most revealing.

Conclusion

Saguaro's intervention raised conversion from 41 to 66 percent, and a design using staggered rollout, a time series and a comparison credit union supports a causal conclusion. Benefits extend to staff cooperation and member experience, with an unintended effect on branches that calls for further work. The intervention should be sustained, adjusted and extended.

References

Armenakis, A. A., & Bedeian, A. G. (1999). Organizational change: A review of theory and research in the 1990s. Journal of Management, 25(3), 293-315. https://doi.org/10.1177/014920639902500303

Golembiewski, R. T., Billingsley, K., & Yeager, S. (1976). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science, 12(2), 133-157. https://doi.org/10.1177/002188637601200201

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.